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Abstract 

Methods  for  fusing  two  computer  vision  methods  are  discussed  and  several  example 
algorithms  are  presented  to  illustrate  the  variational  method  of  fusing  algorithms.  The 
example  algorithms  solve  the  photo-topography  problem:  that  is,  the  algorithms  seek 
to  determine  planet  topography  given  two  images  taken  from  two  different  locations 
with  two  different  lighting  conditions.  The  algorithms  each  employ  a  single  cost 
function  that  combines  the  computer  vision  methods  of  shape-from-shading  and  stereo 
in  different  ways.  The  algorithms  are  closely  coupled  and  take  into  account  all  the 
constraints  of  the  photo-topography  problem. 

One  such  algorithm,  the  z-only  algorithm,  can  accurately  and  robustly  estimate 
the  height  of  a  surface  from  two  given  images.  Results  of  running  the  algorithms  on 
four  synthetic  test  image  sets  of  varying  difficulty  are  presented. 
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Chapter  1 
Introduction 


Over  the  years  computer  vision  researchers  have  developed  algorithms  to  interpret 
grav  scale  and  color  images.  These  algorithms  can  be  used  to  extract  various  types 
of  information  from  the  images  such  as  relative  motion  between  images  (motion- 
vision).  depth  from  a  pair  of  images  (binocular  stereo),  surface  orientation  from  a 
set  of  images  with  different  lighting  (photometric  stereo),  surface  orientation  from  a 
single  image  (shape-from-shading  and  other  shape-from-X  algorithms).  Most  of  these 
algorithms  work  well  in  specialized  circumstances  but  not  as  well  in  general.  They 
are  not  very  robust.  They  work  well  if  everything  is  perfect,  but  don't  do  very  well 
in  sub-optimal  conditions.  They  operate  with  many  restrictive  assumptions,  which 
limit  their  applicability  to  many  real  problems.  A  way  is  needed  to  create  more  robust 
methods  that  can  be  applied  to  wide  range  of  input  images. 

One  obvious  approach  is  to  research  ways  of  enhancing  the  existing  methods  to 
make  them  applicable  to  a  wider  range  of  inputs.  Another  approach  is  to  combine  one 
or  more  methods  to  produce  a  hybrid  method  with  better,  more  robust  performance. 
This  thesis  looks  into  the  latter  approach.  The  resulting  algorithms  are  called  fused 
algorithms.  Fusing  of  two  or  more  computer  vision  methods,  if  they  bring  different 
information  to  the  problem,  can  create  a  more  robust  solution.  It's  like  Kalman 
filtering:  with  more  sensors  you  get  a  better  estimate. 

This  thesis  explores  and  answers  the  following  questions: 

1.  What  are  the  different  fusion  paradigms  and  how  do  they  differ? 

2.  Which  fusion  paradigm  are  best  for  vision  problems? 

3.  How  much  additional  performance  and  robustness  can  be  obtained  via  fusion? 

There  are  two  main  approaches  to  fusing  disparate  algorithms.  I  call  them  the 
module-based  approach  and  the  variational  approach.  In  the  module-based  approach, 
the  fused  algorithms  are  run  separately  on  the  input  images.  The  various  outputs, 
(they  might  be  a  sparse  depth  map  from  binocular  stereo  and  a  dense  surface  ori¬ 
entation  from  shape-from-shading),  are  then  combined  somehow  to  generate  a  single 
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Figure  1-1:  Generic  module-based  fusion  method  flow  chart. 


W»  Al|or*hn>» 


Figure  1-2:  Generic  variational- based  fusion  method  flow  chart. 

solution  (see  Figure  1-1).  The  module-based  approach  is  easy  to  implement  since 
existing  algorithms  can  be  put  together  in  an  ad-hoc  manner  without  having  to  know 
much  about  the  inner  workings  of  each  algorithm.  The  outputs  are  then  combined 
using  physically  derived  constraints  to  generate  the  desired  combined  solution.  The 
approach  can  be  applied  successfully,  but  has  the  disadvantage  that  this  approach 
doesn’t  fully  exploit  the  information  coupling  between  the  methods. 

The  variational  approach,  on  the  other  hand,  closely  couples  the  algorithms  to¬ 
gether  (see  Figure  1-2).  This  is  achieved  by  formulating  a  combined  cost  function 
based  on  the  cost  functions  of  the  separate  algorithms  to  be  fused.  The  result  is  a 
combined  optimization  problem  which  takes  into  account  both  the  explicit  and  im¬ 
plicit  constraints  between  the  methods.  Variational  methods,  by  their  nature,  can 
exploit  any  orthogonalities  (information  content)  in  the  methods.  By  exploiting  all 
the  information,  the  variational  approach  has  the  potential  to  create  robust,  well  per¬ 
forming  combinations  of  algorithms  which  can  be  applied  to  a  wide  range  of  input 
images.  How  to  create  such  algorithms  is  the  focus  of  this  thesis. 


1.1.  PHOTO-TOPOGRAPHY 
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Figure  1-3:  Example  camera  and  light  source  geometry  for  the  two  images  of  photo¬ 
topography. 

1.1  Photo-topography 

A  problem  that  can  benefit  greatly  from  a  fused  vision  algorithm  is  the  problem  of 
photo-topography.  Photo-topography  seeks  to  determine  the  topography  of  a  planet  's 
surface  based  on  two  images  of  the  planet,  taken  from  two  different  vantage  points  at 
two  different  times.  This  situation  is  illustrated  in  Figure  1-3. 

In  some  ways,  this  situation  is  analogous  to  binocular  stereo.  Unfortunately,  the 
images  are  typically  taken  with  two  different  light  source  positions  as  shown  in  the 
figure.  The  two  images  that  result  will,  in  general,  look  quite  different  from  each 
other  even  though  they  are  images  of  the  same  surface  patch.  One  such  set  of  images 
is  shown  in  Figure  9-1  on  page  121. 

Contrast  this  with  the  situation  that  is  normally  true  for  binocular  stereo  images. 
Stereo  image  pairs  are  usually  taken  simultaneously  (or  nearly  simultaneously),  from 
positions  that  are  near  to  each  other,  and  with  the  same  lighting.  The  images  that 
result  look  very  similar  to  each  other  except  for  the  relative  shift  of  objects  (i.e.  the 
disparity)  due  to  their  distance  from  the  cameras.  If  the  disparity  for  all  points  in 
the  images  and  the  relative  geometry  of  the  cameras  is  known,  then  the  depth  of  the 
objects  can  be  computed  directly.  The  images  are  similar,  so  most  stereo  algorithms 
determine  the  disparity  by  trying  to  match  features  in  one  image  with  features  in  the 
other.  Since  the  images  from  photo- topography  look  different  from  each  other,  this 
matching  approach  doesn’t  work. 

Photo- topography  also  shares  some  aspects  with  a  shape- from-shading  problem 
Shape-from-shading  takes  a  gray-scale  image  of  a  surface  and  determines  the  surface 
topography  by  exploiting  the  shading  information  in  the  image.  Shape-from-shading 
requires  that  the  surface  reflectance  properties  be  known.  Assuming  the  reflectance 
properties  of  the  planet's  surface  are  known,  we  could  use  a  shape-from-shading  algo 
rithm  to  estimate  the  surface  topography  from  each  of  the  photo-topography  images. 
Unfortunately,  the  surface  estimates  based  on  each  image  will  typically  be  different. 
They  may  not  even  be  very  similar.  In  the  worse  case,  the  surface  estimates  from 
each  image  may  not  have  the  same  orientation:  one  could  be  concave  while  the  other 
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is  convex.  Thus  planet  photo-topography  is  a  perfect  problem  on  which  to  test  a 
fusion  algorithm  based  on  shape-from-shading  and  stereo. 

Photo-topography  also  has  aspects  in  common  with  a  photometric  stereo  prob¬ 
lem.  Photometric  stereo  uses  two  images  of  the  same  scene  taken  with  two  different 
lighting  conditions.  Both  images  are  from  the  same  vantage  point:  the  camera  is  not 
moved  between  images.  The  result  is  two  images  that  look  different  but  where  the 
correspondence  is  explicitly  known  {a  position  in  one  image  is  always  matched  with 
the  same  position  in  the  other  image).  If  the  light  positions  are  far  enough  apart,  it 
is  possible  to  determine  the  surface  orientation  directly.  For  Lambertian  reflectance, 
two  images  can  constrain  the  surface  orientation  to  two  possible  values  at  each  point. 
With  three  images,  it  is  possible  to  find  a  unique  surface  orientation  at  each  point. 

The  photo-topography  images  have  two  different  light  source  positions,  but  the 
correspondence  is  based  on  binocular  stereo.  Like  photometric  stereo,  the  two  light 
sources  can  constrain  the  surface  orientation,  but  only  if  the  correspondence  is  known. 
Thus  in  photo-topography,  photometric  stereo  and  binocular  stereo  are  closely  linked. 

Currently,  photo- topography  problems  are  solved  using  a  characteristic  strips 
method  [Davis  and  Soderblom,  1984]  or  more  modern  shape-from-shading  meth¬ 
ods  [Van  Hove  and  Carlotto,  1986].  The  characteristic  strips  method  uses  a  general 
method  of  solving  partial  differential  equations  and  is  similar  to  some  early  shape- 
from-shading  algorithms.  In  this  method,  the  solution  is  found  along  strips  starting 
from  a  given  (or  known)  point.  The  solutions  obtained  are  local,  and  may  not  cover 
the  space  of  interest. 


1.2  Related  Research 

This  research  is  most  closely  related  to  the  work  of  Horn  [ikeuchi  and  Horn.  1981]. 
[Horn  and  Brooks,  1986],  [Horn,  1986],  [Horn,  1989],  Gennert  [Gennert,  1987],  and 
Szeliski  [Szeliski,  1990],  [Szeliski.  1991].  The  variational  (least  squares)  approach  I 
have  taken  is  based  significantly  on  the  work  of  Horn  [Horn,  1989],  [Horn  and  Schunk, 
1981  ],  [Negahdaripour  and  Horn,  1985]  and  on  insights  gained  from  my  background  in 
control/estimation.  The  shape-from-shading  part  of  this  thesis  builds  upon  the  work 
of  Horn  [Horn,  1989],  Szeliski  [Szeliski,  1991],  and  Leclerc  and  Bobick  [Leclerc  and 
Bobick.  1991].  The  stereo  part  of  this  thesis  builds  on  the  gray-scale  stereo  algorithm 
of  Gennert  [Gennert,  1987].  I  use  the  hierarchical  basis  functions  of  Szeliski  and  use 
conjugate  gradient  optimization,  as  do  Leclerc  and  Bobick  [Leclerc  and  Bobick,  199l]. 

This  research  is  also  related  to  the  work  of  Hartt  and  Carlotto  [Hartt  and  Carlotto. 
1989],  [Heipke,  1992],  Wildey  [Wildey,  1973],  and  McEwen  [McEwen.  1985]  in  that 
they  try  to  solve  the  same  problem;  that  is,  the  determination  of  planet  topography 
from  a  pair  of  images. 

Hartt  and  Carlotto  use  a  Bayesian  formulation  with  multiple  images  that  in  some 
ways  is  very  similar  to  the  methods  I  use.  Their  cost  function  and  their  approach  is 
however  very  different.  The  Bayesian  formulation  leads  them  to  a  cost  function  of 
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the  form  (using  my  notation) 

C(  =  )  =  JJ  Y,\E(k)(^y)  -  Rk(p,q)\  +  X\V=(x.y)\2dxdy.  (1.1) 

While  they  discuss  methods  of  estimating  the  albedo  of  the  surface,  -esults  are  shown 
for  constant  albedo  surfaces  with  Lambertian  reflectance.1 

They  solve  their  problem  using  a  Markov  Random  Field  Model  via  a  Gibbs  distri¬ 
bution  [Geman  and  Geman,  1984].  They  use  the  Metropolis  optimization  algorithm 
[Metropolis  et  al. ,  1953]  with  a  coarse- to- fine  multi-resolution  strategy  to  obtain  their 
results.  They  show  results  for  the  algorithm  on  multiple  (2)  images  based  on  a  digital 
elevation  map  of  upstate  New  York.  They  obtain  better  estimates  when  two  images 
are  used  instead  of  one.  It  is  hard  to  determine  from  the  results  they  show,  but  it 
appears  that  their  estimates  contain  quite  a  bit  of  error  (only  the  estimated  images 
are  shown).  The  efficiency  of  their  algorithm  is  also  hard  to  gauge.  They  state  in 
their  paper  that  approximately.  200  sweeps  of  each  image  is  performed  at  each  reso¬ 
lution.  Inferring  that  they  used  60-by-60  pixel  images,  with  3  resolution  levels,  and 
each  sweep  evaluated  the  cost  function  n  times  where  n  is  the  number  of  pixels  on 
that  level,  the  number  of  function  evaluations  they  required  is  approximately 

.V  %  200  *  (602  +  302  +  152)  (1.2) 

s=  945000  function  evaluations.  (1.3) 

This  is  significantly  more  function  evaluations  that  the  algorithms  I  have  developed 
(my  algorithms  typically  require  at  most  1500  function  evaluations  for  65-bv-65  pixel 
images  and  much  less  on  some  images). 

Heipke  discusses  the  solution  to  a  multiple  image  photo- topography  problem.  In 
the  paper  [Heipke,  1992],  he  states  that  he  uses  a  least  squares  cost  function.  In 
addition,  the  constraint  equations  that  he  develops  are  the  same  as  mine  except  that 
they  apply  to  more  than  two  images.  Since  Heipke  is  not  very  explicit  in  this  paper,  it 
is  difficult  to  tell  exactly  how  he  solves  the  problem.  His  results  are  encouraging  but 
also  show  that  his  method  is  very  sensitive  to  noise  in  the  images.  Heipke’s  algorithm 
shares  many  things  in  common  with  the  algorithms  I  develop  in  this  thesis  since  he 
starts  with  the  same  constraint  equations.  Even  so.  his  algorithm  doesn't  perform  as 
well  as  my  algorithms. 

The  work  of  McEwen  is  based  on  traditional  photoclinometry  methods  and  is 
thus  profile  based.  In  [McEwen,  1985],  McEwen  uses  information  from  two  images  of 
the  same  location  under  different  lighting  conditions  to  distinguish  between  albedo 
and  reflectance  effects.  The  image  sets  McEwen  uses  are  taken  from  nearly  the  same 
location  so  that  the  pixel  correspondence  is  easy.  In  many  ways,  McF.wen's  methods 
parallel  photometric  stereo  much  more  than  my  algorithms. 


‘I  make  similar  assumptions  for  most  of  this  thesis 


•  •  •  • 


•  •  • 
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Wildey  basically  solves  a  stereo  problem  [Wildey.  1973].  He  assumes  that  that 
the  light  source  is  in  the  same  position  for  each  image  and  tries  to  determine  pixel 
correspondence  by  matching  image  brightness.  His  method  estimates  the  topography 
of  the  planet  surface  as  it  proceeds.  He  mentions  an  enhancement  to  this  method 
that  would  take  into  account  the  estimated  brightness  of  the  reconstructed  surface. 
This  enhancement  is  closely  related  to  my  algorithms  (and  shape-from-shading)  and 
would  be  applicable  to  images  with  similar  lighting. 

Another  related  paper  is  the  shape-from-shading  and  stereo  fusion  algorithm  of 
Grimson  [Grimson,  1984].  In  that  paper,  Grimson  describes  a  method  for  enhancing 
the  surface  reconstruction  step  required  after  feature-based  stereo  using  shading  in¬ 
formation.  His  method  uses  the  shading  information  only  along  features  to  constrain 
the  surface  reconstruction  and  is  loosely  based  on  the  methods  of  photometric  stereo 
[Woodham.  1980]. 

1.2.1  Relation  of  this  thesis  to  sensor  fusion 

The  goal  of  this  thesis  is  to  investigate  ways  of  combining  or  fusing  two  different 
computer  vision  methods.  Thus  this  thesis  is  related  to  sensor  fusion  techniques  (also 
called  data  fusion  or  information  fusion  in  some  contexts).  Sensor  fusion  is  a  way  of 
combining  the  information  from  multiple  sensors  to  create  better  parameter  estimates 
than  can  be  achieved  with  the  individual  sensors  alone. 

Fusion  has  a  much  longer  history  than  its  use  in  vision  algorithms.  Researchers 
and  engineers  have  been  combining  disparate  sensors  to  obtain  better  estimates  of 
the  outside  world  for  a  long  time.  In  the  literature,  three  different  approaches  can 
be  identified:  1)  Fusion  via  estimation  theory.  2)  Fusion  via  decision  theory,  and  2) 
fusion  via  artificial  intelligence  methods. 

The  traditional  approach  involves  the  use  of  Kalman  Filtering  [Gelb.  1974].  In 
Kalman  filtering,  or  estimation  as  it  more  broadly  called,  the  sensor  physics  are  an¬ 
alyzed  and  a  noise  model  is  postulated.  Based  on  this  information,  a  cost  function 
is  formulated.  Typical  cost  functions  are  based  on  maximum  likelihood  estimation, 
maximum  a  priori  estimation,  minimum  variance  estimation,  or  least  squares  estima¬ 
tion.  For  simple  noise  models  (such  as  Gaussian  white  noise),  these  cost  functions 
have  the  form, 

inin  J  =  J  L(x - )  (1.4) 

possibly  subject  to  additional  constraints  imposed  by  the  physics.  L(x, . . .)  is  some 
non-linear  functional.  This  same  type  of  cost  function  is  formed  when  solving  vision 
problems  using  the  variational  approach  [Horn,  1989].  Estimation  theory-based  fusion 
includes  methods  that  rely  on  Bayesian  statistics  [Lee,  1990],  [Richardson  and  Marsh. 
1988],  [Hung  et  a/.,  1988],  as  well  as  least-squares  approaches  [Shaw  ef  a/.,  1988]. 

Decision  theory  approaches  use  Bayesian  Reasoning  [Thomopoulos,  1989],  Shafer- 
Dempster  Reasoning  [Bogler,  1987]  or  ad  hoc  methods  [Mitiche  and  Aggarwal,  1986]. 
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The  artificial  intelligence  methods  range  from  rule-based  hypotheses  [Belknap  et  al.. 
1986],  [Belknap  et  al..  1986],  [Nandhakumar  and  Aggarwal,  1988].  to  blackboard 
schemes  [Harmon  et  al..  1986],  to  knowledge  representation  approaches  [Pan.  1989]. 
to  artificial  neural  networks  [Brown  et  al..  1991].  Sensor  fusion  is  an  important 
problem  for  mobile  robots  and  military  threat  detection  as  evidenced  by  the  above 
references. 

An  important  problem  in  sensor  fusion  that  is  not  addressed  in  this  thesis  is  the 
fusion  of  widely  differing  sensors.  The  problem  is  particularly  difficult  for  mobile 
robots  where  the  sensors  may  provide  information  at  several  conceptual  levels.  For 
example,  sensors  may  be  available  for  wheel  rotation,  range  images,  visual  images, 
touch  and  sound.  Combining  information  from  such  disparate  sources  is  much  more 
difficult  than  combining  information  from  sources  that  operate  on  the  same  level  (such 
as  combining  infrared,  range  and  visual  images). 

1.2.2  Relation  of  this  thesis  to  vision  fusion  schemes 

The  computer  vision  literature  is  roughly  segmented  along  module  boundaries.  These 
modules  are  related  to  our  current  ideas  about  how  human  vision  works.  For  in¬ 
stance  there  are  algorithms  for  binocular  stereo,  motion  vision,  and  a  whole  host 
of  shape-from-X  algorithms.  Each  of  these  algorithms  performs  reasonably  well  in 
ideal  conditions  but  can  degrade  quickly  in  typical  real-world  situations.  It  has  been 
suggested  by  many  researchers  [Aloimonos  and  Basu.  1988].  [Waxman  and  Duncan. 
1986]  that  a  combination  of  these  modules  will  do  a  better  job  of  estimating  the  ex¬ 
ternal  world.  Of  course,  these  ideas  are  firmly  supported  by  estimation  theory  where 
it  can  be  easily  shown  that  adding  sensors  cannot  degrade  an  estimate  and  most  often 
enhances  it. 

Along  these  lines,  some  researchers  in  the  computer  vision  community  have  been 
looking  into  ways  of  combining  more  than  one  vision  cue  (and  associated  algorithm) 
in  order  to  obtain  either  better  or  more  robust  estimates  of  the  external  world.  The 
techniques  for  combining  the  methods  range  from  those  that  use  Bayesian  estima¬ 
tion  theory  [Matthies  and  Elfes.  1988],  [Hartt  and  Carlotto.  1989].  to  those  that  use 
module-based  methods  [Moerdler  and  Render.  1987],  [Moerdler  and  Boult.  1988], 
[Grimson.  1984].  Another  common  technique  is  to  use  an  analytical  approach  [Wax- 
man  and  Duncan,  1986],  [Aloimonos  and  Basu.  1988],  [Hu  and  Shrikhande.  1990], 
relying  on  the  constraints  from  the  fused  cues  (under  particular  assumptions)  to  gen¬ 
erate  either  a  unique  or  a  finite  number  of  possible  solutions. 

I  prefer  the  estimation-based  methods,  since  all  the  information  available  can  be 
exploited,  and  the  assumptions  behind  a  particular  method  can  be  quantified.  These 
methods  also  rest  on  a  firm  foundation.  The  module-based  methods,  in  contrast, 
are  more  ad  hoc.  The  analytical  approach  can  be  used  within  the  estimation-based 
approach  to  constrain  the  solution. 
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1.2.3  Relation  of  this  thesis  to  photoclinometry 

One  of  the  reasons  for  this  research  is  to  improve  methods  of  planetary  mapping.  In 
particular,  this  research  aims  to  widen  the  applicability  of  computer  methods  to  this 
difficult  task.  Current  techniques  for  planetary  mapping  work  either  on  symmetric 
objects  or  along  ridges  [Davis  and  Soderblom,  1984],  or  use  the  same  methods  as 
shape-from-shading  [Wildey,  1975],  [Van  Hove  and  Carlotto.  1986],  [Kirk,  1987]  al¬ 
though  the  theory  was  developed  independently  within  the  geophysics  community. 
In  that  community,  shape-from-shading  is  called  two-dimensional  photoclinometry. 
Important  aspects  for  photoclinometry  are  the  determination  of  the  true  reflectance 
function  for  a  planet  [Davis  and  McEwen,  1984],  [Wilson  et  al. ,  1985],  [McEwen. 
1991],  [Helfenstein  et  al.,  1991]  and  how  to  deal  with  non-constant  albedo  [Davis  and 
Soderblom.  1984],  [Helfenstein  et  al.,  1991]. 

While  there  is  no  doubt  that  using  the  correct  reflectance  function  and  accounting 
for  non-constant  albedo  is  important,  most  of  the  algorithms  in  this  thesis  are  for  the 
simplified  case  of  Lambertian  reflectance  and  constant  albedo.  Both  of  these  simpli¬ 
fications  are  not  fatal:  the  algorithms  can  be  easily  generalized  to  other  reflectance 
functions  and  the  non-constant  albedo  case  is  dealt  with  in  Section  8.1.  In  fact,  the 
algorithms  developed  will  work,  without  change,  with  any  reflectance  function  that 
is  smooth  and  that  doesn’t  contain  multiple  extrema. 


Chapter  2 
Background 


2.1  Coordinate  Systems 

The  solution  to  a  photo-topography  problem  is  a  description  of  the  surface  topog¬ 
raphy.  The  most  straightforward  description  for  the  surface  is  to  represent  it  as  a 
height  function  over  some  suitable  2-D  domain.1  that  is 

z  =  z(x,y).  (2.1) 

Within  the  vision  literature,  there  are  two  different  choices  for  the  2-D  domain.  The 
image-centered  domain  uses  the  coordinates  of  the  image  as  the  fundamental  do¬ 
main  and  assigns  a  depth  (or  height)  value  to  the  surface  point  that  projects  to  each 
image  position.  If  orthographic  projection  is  used,  then  the  projection  mapping  is 
straightforward.  However  if  perspective  projection  is  used,  then  this  mapping  can  be¬ 
come  quite  complex  (especially  for  surface  normal  calculations).  The  object-centered 
domain  uses  a  coordinate  system  associated  with  the  object  and  assigns  a  surface 


‘Also  known  as  a  Monge  patch 


Object-Centered  Coordinate  System 


Figure  2-1:  Coordinate  system  choices. 
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height  value  to  each  point  in  the  domain  (see  Figure  2-1).  Projecting  points  on  the 
object  to  points  in  the  image  is  straightforward  using  this  representation  for  either 
orthographic  or  perspective  projection.  However,  the  projected  points,  won't  in  gen¬ 
eral  map  to  the  center  of  each  pixel.  To  obtain  values  at  each  pixel  then  requires 
some  type  of  interpolation. 

In  a  stereo  system,  there  is  also  the  choice  of  whether  the  coordinate  system  should 
favor  one  image  or  the  other.  Since  there  is  no  apriori  reason  to  believe  that  one  image 
has  better  information  than  the  other  image,  it  is  best  to  choose  a  neutral  coordinate 
svstem  rather  than  risk  biasing  the  result  toward  one  or  the  other  image. 


2.2  Image  Generation  Process 

The  photo-topographv  robi  m  is  basically  an  inverse  problem.  We  seek  to  determine 
the  topography  of  the  object  that  created  the  images  at  hand.  Those  images  are  based 
on  the  object  (a  forward  problem).  That  is,  the  interaction  of  light  with  an  object, 
as  seen  by  the  viewer  creates  the  image.  The  physics  behind  this  process  are  detailed 
in  the  sections  that  follow. 

The  image  generation  process  can  be  neatly  broken  into  four  stages  (see  also 
Figure  2-2), 

1 .  light  falls  upon  an  object  (object  irradiance) 

2.  the  light  interacts  with  the  object  and  is  re-emitted  or  reflected  (object  radiance) 

3.  the  light  then  travels  to  the  viewer  where  it  is  projected  onto  the  image  plane, 
(image  projection  and  image  irradiance) 

4.  the  light  is  absorbed  by  the  material  of  the  image  plane  and  converted  into  some 
signal  that  can  be  sensed  (image  transduction). 


*j> 
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2.2.1  Object  irradiance 

The  light  that  falls  on  a  particular  patch  of  an  object  depends  on  the  properties  of  the 
light  sources  that  are  visible  (i.e..  unobstructed)  from  that  patch.  Among  possible 
light  sources  are  point  sources,  distributed  sources,  and  other  surface  patches  (e  g., 
interflection).  At  each  point,  £,  on  the  surface,  and  for  each  direction,  s.  the  irradiance 
distribution  function,  E(£;»)  [radiance/solid  angle]  captures  this  information.  For 
instance,  the  distribution  function  for  a  single  point  source,  ignoring  interflection,  is 
given  by 

£({;§)  =  A6(s,So)  (2.2) 


•  • 


•  • 
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Figure  2-2:  Image  generation  process  steps. 

where  A  is  the  light  intensity.  So  is  the  unit  vector  in  the  light  source  direction,  and 
6  is  a  direction  vector  version  of  the  Dirac  delta2 

6(t.t0)  =  O.when  t  ^  t0. 

JJs  f(i)6(i'i0)dt  =  /(t°),  ("'3) 

where  the  integral  is  over  the  surface  of  the  sphere  S. 

2.2.2  Object  Radiance 

The  light  that  is  emitted  by  a  surface  depends  both  on  the  light  that  impinges  on 
the  surface  and  the  surface  reflectance  properties.  The  irradiance  distribution  func¬ 
tion.  E(C-  s)  captures  all  the  needed  information  about  the  light  source,  while  the 
surface  reflectance  properties  can  be  described  using  the  Bidirectional  Reflectance 
Distribution  Function  (BRDF).  The  BRDF  is  an  intrinsic  function  of  the  surface  and 
doesn't  depend  on  surface  irradiance.  At  the  point  £.  the  BRDF  /(^:n:  v:s)  relates 
the  brightness  of  the  surface  patch  with  normal  n  illuminated  from  the  direction  s 
and  as  seen  from  the  direction  v.  Using  these  two  distribution  functions,  the  surface 
radiance  [power/solid  angle/area]  in  the  direction  v  can  then  be  defined  as 

L(£;n;v)=//  /({;n:  v:s)£(£:s)(s  •  n)<L;(s)  (2.4) 


2This  delta  function  is  defined  on  the  surface  of  the  sphere.  One  way  to  define  it  is  to  repre¬ 
sent  the  direction  t  in  spherical  coordinates,  t  =  (sin0cos®,sin0sin$>,cos0)T,  so  that  we  require 
f-i%  JT  Hi(<P,9)Ao(<Po.8o))i°*9d<pd6  =  1,  hence, 

6(t(®,  6).  to(®o.  90))  —  6(0  —  ©0 )6(0  —  9o)/cos(6q), 

where  6(  .)  is  the  normal  scalar  Dirac  delta. 
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where  H( n)  is  the  hemisphere  of  possible  light  source  directions  for  the  patch  at 
surface  point  (  with  normal  n.  and  duj(s)  is  the  solid  angle  subtended  in  the  direction 
s.  Representing  s  in  spherical  coordinates  this  expression  becomes 

I(£;n;v)  =  y  J  '  /((;n;  v;s)£(f;s)sinflcos0d0</0.  (2.5) 

The  two  most  common  reflectance  models  are  the  Lambertian  model  for  matte 
surfaces,  and  the  specular  model  for  shiny  metallic  surfaces.  In  the  Lambertian  case, 
the  BRDF  is  independent  of  the  light  source  direction,  normal  direction  and  viewing 
direction, 

/({;n;v;s)  =  -p({)  (2.6) 

7T 

where  p(£)  is  the  surface  albedo,  or  the  fraction  of  light  re-emitted  by  the  surface. 
Evaluating  Equation  2.5  for  a  Lambertian  surface  illuminated  by  a  single  point  source 
at  infinity,  the  surface  radiance  is  found  to  be 


£({;n;v)  =  ^(s0-n). 

JT 


(2.7) 


In  the  specular  case,  all  the  light  from  the  direction  s0  is  reflected  into  the  direction 
2(n  •  s0)n  —  s0  so  the  BRDF  is 


/(£: n;v;s)  =  />(0<5(v,2(n  •  s0)n  - 


(2.8) 


For  a  single  point  source  at  infinity,  the  surface  radiance  for  a  such  a  surface  is  found 
to  be 

L{£;  n;v)  =  Ap(f)6(v,2(n  ■  s0)n  -  s0)  (2.9) 

In  general,  the  albedo  of  a  Lambertian  surface  is  different  from  the  albedo  of  a  specular 
surface. 

See  [Wilson  et  a/.,  1985],  [McEwen.  1985],  and  [Davis  and  McEwen,  1984]  for  a 
sampling  of  the  types  of  radiance  functions  used  within  the  photochnometry  field. 


*) 


2.2.3  Reflectance  Map 

The  preceding  representation  of  the  surface  radiance  is  a  local  representation  in  that 
the  radiance  is  defined  based  on  local  surface  orientation,  viewer  direction  and  light 
source  direction.  Many  vision  researchers  have  found  it  convenient  to  use  a  represen¬ 
tation  of  the  surface  radiance  based  on  a  global  coordinate  system.  This  alternate 
representation  is  called  the  Reflectance  function.  Given  known  surface  properties  and 
a  known  light  source  distribution,  the  reflectance  function,  R(£,  n;  v),  is  defined  using 
the  corresponding  surface  radiance  via  a  change  in  coordinates, 

#(fc;nc;vc)  =  Itf;n;v)  (2.10) 


(2.10) 
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Figure  2-3:  Two  Perspective  projection  geometries. 

where  n c •  and  Vq  are  the  surface  position,  normal  vector,  and  viewing  direction 

in  global  coordinates. 

2.2.4  Image  Projection 

Points  in  the  image  plane  are  related  to  points  on  the  object  via  perspective  projection. 
Figure  2-3  shows  two  perspective  projection  geometries.  Figure  2-3(a)  shows  the 
projection  geometry  for  a  camera  where  the  image  plane  is  behind  the  lens.  The 
projection  of  the  center  of  the  lens  into  the  image  plane  is  called  the  principal  point  .3 
Choosing  the  origin  of  the  image  coordinate  system  to  be  at  the  principal  point  leads 
to  the  simple  equations  presented  in  this  section.4  Note  that  objects  are  projected 
onto  the  image  plane  in  an  upside  down  orientation. 

An  equivalent  geometry  is  shown  in  Figure  2-3(b).  The  only  difference  between 
the  camera  geometry  shown  in  Figure  2-3(a)  and  this  geometry  is  that  the  image  plane 
has  been  moved  to  in  front  of  the  lens.  While  this  geometry  is  not  physically  possible. 


3The  center  of  projection  is  at  the  back  nodal  point  for  thick  lenses 

’’Determining  the  position  of  the  principal  point  for  a  given  camera  is  part  of  the  classic  interior 
orientation  problem  (see  (Horn.  1986]) 
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it  produces  exactly  the  same  image  (as  can  be  seen  by  similar  triangles)  except  that 
the  image  is  no  longer  upside  down.  Mathematically,  the  difference  between  these  two 
geometries  is  just  a  sign  change  in  the  equations.  For  instance,  the  image  position  r 
of  the  object  point  R  in  Figure  2-3(a)  is 


r 

r  z 


R  z 


(2.11) 


while  it  is 


r 

r  z 


R  z 


(2.12) 


for  the  geometry  in  Figure  2-3(b).  These  are  Perspective  Projection  Equations.  In 
both  expressions,  z  is  a  unit  vector  in  the  direction  of  the  negative  optical  axis.  In 
the  equations  that  follow,  the  geometry  of  Figure  2-3(b)  will  be  used  to  avoid  having 
to  keep  track  of  the  minus  sign  in  the  perspective  projection  equation. 

The  equations  above  can  be  simplified  if  a  special  coordinate  system  is  used, 
namely  a  coordinate  system  with  origin  at  the  principal  point,  and  orientation  such 
that  the  x  and  y  axes  are  aligned  with  the  image  coordinates  and  with  the  negative 
;  axis  along  the  optical  axis  of  the  lens.  This  coordinate  system  is  evident  in  the 
labels  assigned  to  the  axes  in  Figure  2-3(b).  For  this  coordinate  system,  r  z  =  /  (a 
negative  number).  Expanded,  these  equations  are 


(2.13) 


x/f  \ 

(  x/z 

y/f  = 

Y/Z 

f/f  ) 

{  z/z 

when  r  =  {x.y.  f)T  and  R  =  (X.  Y.  Z)T . 


2.2.5  Orthographic  Projection 

The  projection  equations  can  be  further  simplified  when  the  depth  range  of  the  object 
is  small  compared  to  the  distance  of  the  object  from  the  camera.  The  resulting 
approximation  is  called  Orthographic  Projection.  Consider  the  first  order  Taylor's 
series  expansion  of  the  perspective  projection  equation  about  a  nominal  depth  Z0. 

7  =  5-  +  -  Zo)  +  0((Z  -  Z0)2)  (2.14) 

where  R  =  (X,  Y.  Zo)T.  When  (Z  —  Zo)  4Z.  Zo.  the  first  order  term  can  be  neglected 
resulting  in  the  orthographic  approximation. 
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Orthographic  projection  is  convenient  for  computer  vision  problems  since  the  map¬ 
ping  is  linear.  In  fact,  most  shape- from-shading  algorithms  assume  orthographic 
projection.  Unfortunately,  the  full  perspective  projection  must  be  used  for  photo¬ 
topography  to  avoid  throwing  out  the  very  information  we  wish  to  estimate  (i.e..  the 
stereo  depth  information). 


2.2.6  Image  Irradiance 


The  mapping  between  object  points  and  image  points  is  only  half  of  the  story  for 
image  generation.  We  also  need  to  know  how  the  brightness  of  the  image  is  affected 
bv  the  lens.  Figure  2-4  shows  that  an  object  patch  of  area  60  projects  to  an  image 
patch  of  area  61.  Assuming  a  perfect  lens,  the  image  irradiance  [power/area]  from 
this  patch  is 


Ei  r)  =  ) 

*  \J  / 


COS  Q 


(2.16) 


where  I(£|  r):  n:  v(r))  is  the  radiance  of  the  corresponding  object  patch,  d  is  the 
diameter  of  the  lens,  and  a  is  the  off-axis  angle  of  the  projecting  ray.  In  this  equation, 
v  and  £  are  functions  of  r  via  the  projection  from  image  points  r  to  object  points. 
Equivalently,  this  expression  can  be  restated  using  the  reflectance  function  rather 
than  the  radiance  function. 


£tr)  =  /?(s(r):  n:  vfr))-^ 

4 


cos4  Q 


(2.1 


i  i 


0 


2.2.7  Image  Transduction 

The  final  stage  in  producing  an  image  is  the  conversion  of  light  into  a  signal  that  can 
be  used,  namely  digital  form.  Whether  the  digital  images  are  scanned  photographs 
or  are  obtained  directly  via  a  digital  camera,  they  will  contain  distortion  (either 
spatially  or  in  color).  The  effect  of  this  distortion  can  be  removed  via  calibration 
so  that  the  measured  image  irradiance  can  be  related  to  the  object  radiance  in  a 


•  •  • 
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« 


Figure  2-5:  Stereo  geometry. 

straightforward  way.  With  perfect  calibration,  the  image  irradiance  equation  can  be 
put  in  the  especially  simple  form, 

£(r)  =  /?({(r);  n;v(r))  (2.18) 

where  the  cos4  a  and  constant  scale  factors  have  been  removed  as  part  of  the  calibra 
tion.  Equation  2.18  is  referred  to  as  the  Image  Irradiance  Equation. 


2.3  Stereo 

The  images  in  a  photo-topography  problem  are  taken  from  two  different  vantage 
points  (see  Figure  2-5).  If  the  relative  position  of  the  cameras  is  known,  and  it  is 
known  which  pixels  in  the  left  image  correspond  to  which  pixels  in  the  right  image, 
then  it  is  possible  to  determine  the  depth  directly  for  the  surface  points  that  map 
to  those  pixels.  This  is  the  basis  of  binocular  stereo.  Of  course,  the  hard  part  is 
determining  the  correspondence  between  pixels. 

For  normal  stereo  situations,  the  cameras  are  close  together  and  both  pictures 
are  taken  simultaneously.'  The  stereo  images  that  result  look  very  similar,  mostly 
differing  in  the  shift  of  objects  in  each  image  caused  by  perspective  projection.  The 
difference  in  the  shift  of  an  object  point  in  the  left  image  and  the  right  image  is  called 
the  disparity.  The  photo- topography  images  are  take:  wth  c  ameras  that  are  often  far 


’The  stereo  images  of  aerial  photography  are  taken  neans  set.  tanrously. 
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apart  and  at  different  times.  In  this  case,  it  is  possible  (because  of  differing  lighting) 
for  the  two  images  to  look  very  different.  This  makes  the  correspondence  problem 
even  harder. 

To  see  how  the  depth  can  be  determined  directly  from  the  disparity  consider 
Figure  2-5.  Points  in  each  image  map  to  rays  in  3-space.  If  corresponding  points 
in  each  image  map  to  rays  that  intersect,  the  depth  can  be  determined  using  simple 
geometry.  If  the  rays  don't  intersect,  we  can  use  geometry  to  determine  the  depth  of 
the  point  that  is  halfway  between  the  rays  at  their  closest  approach. 

First,  find  the  relationship  between  the  two  camera  coordinate  systems.  Suppose 
we  know  the  position  of  the  principal  point  of  each  camera  in  some  global  coordinate 
svstem.  Pi  and  P2,  and  we  also  know  the  rotational  transformation  matrices  from 
each  local  camera  coordinate  system  to  the  global  coordinate  system.  7\  and  T2.  then 
the  coordinates  of  the  point.  in  the  two  camera  coordinate  systems  is 

Rl  =  Trlu-pl) 

R2  =  T;l(t  -  Pa).  (~19) 

Equations  2.19  are  the  Stereo  Constraint  Equations  which  relate  the  position  of  points 
in  the  global  coordinate  system  to  points  in  each  of  the  local  camera  coordinate 
systems. 

By  removing  £  from  the  above  equations  and  defining  b  =  P2-Pi-  the  relationship 
between  a  point  in  Camera  Coordinate  System  1  and  a  point  in  Camera  Coordinate 
Svstem  2  is  found  to  be 

R,  =  T^'b  +  T-'TiRi.  (2.20) 

Now.  determine  the  relationship  between  disparity  and  depth.  Suppose  we  are 
given  image  points,  iq  =  (x^yi ,})T  and  r2  =  (-r2. 1/2- /)T  (one  in  each  image),  that 
correspond  to  the  same  surface  point,  then  the  best  estimate  for  the  surface  position 
can  be  found  by  finding  the  point  on  each  ray  (along  ri  and  r2)  where  the  distance 
between  the  rays  is  minimized.  That  is,  the  problem 

min  ||-sri  +  b  +  tr2||2  (2.21) 

9.1 


must  be  solved  where  s  and  t  are  scalar  parameters.  For  now  assume  all  the  vectors 
are  given  on  the  same  basis. 

Bv  differentiating  the  above  equation  with  respect  to  s  and  t.  setting  the  resulting 
equations  to  zero,  and  solving,  it  is  found  that  the  minimum  occurs  when 


(r2  •  r2)(b  •  Ti)  -  (i*i  ■  r2)(b  •  r2)  _  (r2  x  b)  •  (r2  x  rt) 
(ri  •  ri)(r2  •  r2)  -  (ri  •  r2)J  ||r2  x  rj||2 

(£i  •  r2)(b  •  t-t)  -  (rt  •  rt)(b  ■  r2)  _  (rt  x  b)  •  (r2  x  n) 
(ri  •  r,)(r2  •  r2)  -  (ri  ■  r2)2  ||r2  x  ri||2 


(2.22) 
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If  rj.  r2,  and  b  are  coplanar  then  the  above  set  of  equations  is  just  a  fancy  way  of 
writing  the  Law  of  Sines. 

The  global  position  of  the  point  halfway  between  the  two  rays  at  the  closest 
approach  is  then 


£  as  Pj  +  srj  +  -(-sr,  +  b  +  tri)- 

n  ,  b  1  (J2r2  x  b)  ■  (T2r2  x  T,r,)^_ 
=  ri  +  —  +  - - —  — - ~2 - 1  lfl 

2  2  ||T2r2  x  r,r,|| 


1  (Tjr,  x  b)  •  (T2r2  x  r,ri)^  _ 

+  — — ~ — - ^ - -*  2^*2 

2  ||T2r2  x  7jri||2 


(2.23) 


(2.24) 


where  Equation  2.24  has  been  written  for  Pj,  P2,  and  b  given  in  global  coordinates, 
and  rj.  r2  given  in  the  appropriate  local  camera  coordinate  system. 

The  equations  simplify  greatly  if  the  rt,  r2,  and  b  are  coplanar  (i.e.,  if  (r2xri)  b  = 
0),  the  cameras  are  aligned  so  that  the  optical  axes  are  in  the  direction  — z,  and  x  is 
along  b  (y  is  chosen  to  complete  the  right-handed  coordinate  system).  This  is  exactly 
the  stereo  geometry  that  is  assumed  to  exist  for  most  binocular  stereo  algorithms. 
With  these  restrictions,  the  equation  above  becomes 


P.  +  5  + 


*>(ri  +  r2) 
2(zi  -  x2)‘ 


(2.25) 


where  ri  =  (ii,yi,/)r,  r2  =  (x3,y 2,/)T,  and  b  =  (6,0.0)T.  The  quantity  (x2  -  xj 

in  the  above  equation  is  the  disparity  mentioned  earlier.  In  this  form,  the  depth  r  is 

found  to  be  b  bf 

z  =  t  ■  z  =  P,  •  z  +  -  +  ■  f-  (2.26) 

l  (x2  —  x2) 

Note  that  the  disparity  can  be  mapped  directly  into  depth  only  in  this  special  situa¬ 
tion.  For  more  general  situations.  Equation  2.24  must  be  used. 


2.4  Photo-topography  Problem  Formulaton 

We  now  have  enough  background  to  formulate  the  photo- topography  problem.  The 
problem  to  be  solved  is: 

Given  two  images  of  an  area  on  the  planet's  surface,  taken  at  two  differ¬ 
ent  times  from  two  different  positions,  determine  the  topography  of  that 
section  of  the  planet’s  surface. 

The  solution  is  constrained  by  geometry  and  the  image  generation  process.  Specifi¬ 
cally  each  image  is  constrained  by  the  perspective  projection  equation  (Equation  2.12), 
the  image  irradiance  equation  (Equation  2.18)  and  the  stereo  constraint  equations 
(Equations  2.19). 


2.4.  PHOTO-TOPOGRAPHY  PROBLEM  FORMVLATOS 


:J1 


Combining  these  equations  we  find  that  the  photo-topographv  problem  is  constr¬ 
ained  such  that 


£(1»(r1)  =  ^(1)U:n;-r,r1) 
£<2>(r2)  = 


where 


ri 


Ti 


fTcHi-Px) 

TrlU-Pi)-Ttxt‘ 

fTr'(z-pt) 

T^(Rg~P2)T,z2' 


(2.28) 


are  the  local  image  plane  position  vectors  (one  in  each  camera  coordinate  system). 
£*”  and  £<2)  are  the  image  brightness  measured  in  the  first  and  second  cameras 
respectively,  and  i?*1*  and  R *2'  are  the  reflectance  maps  based  on  the  first  and  second 
light  source  positions.  As  before.  £,  is  the  surface  position  in  global  coordinates. 

Equations  2.27  and  2.28  are  the  General  Photo-topography  Constrain 1  equations. 
They  state  that  in  the  absence  of  noise,  the  photo-topographv  images  £(,)  are  created 
by  the  interaction  of  light  with  the  underlying  surface  z(£)  via  the  reflectance  maps 
and  as  seen  by  the  two  cameras.  Note  that  these  equations  can  be  used  to 
generate  a  set  of  photo-topography  images  given  the  surface  description.  c(£).  In 
contrast,  determining  the  surface  topography  given  the  images,  camera  geometry  and 
reflectance  properties,  is  an  inverse  problem. 

It's  instructive  to  review  the  assumptions  behind  these  equations.  The  perspec¬ 
tive  projection  equations  assume  perfect  lenses  and  perfect  knowledge  of  the  camera 
principal  points  and  optical  axes.  The  surface  radiance  equation  assumes  we  have 
perfect  knowledge  of  the  surface  reflectance  properties,  light  source  directions,  and 
all  surface  points  are  visible  from  both  cameras  (i.e..  there  are  no  self  occlusions). 
The  simplified  form  of  the  image  irradiance  equation  assumes  we  either  have  a  per¬ 
fect  sensor  or  we  can  perfectly  calibrate  the  sensor  to  remove  any  abnormalities  from 
the  sensor/lens  combination.  The  stereo  equations  assume  we  know  the  relative  posi¬ 
tion  and  orientation  of  the  two  cameras  perfectly.  The  only  assumption  that  is  truly 
artificial  is  the  assumption  of  perfect  knowledge  of  the  reflectance  maps.6  With  more 
careful  measurements  and  more  expensive  equipment  it  is  possible  to  approach  per¬ 
fect  knowledge  of  the  other  assumptions.  The  assumption  that  all  surface  points  be 
visible  merely  restricts  the  bumpiness  of  the  surface  that  this  research  is  applicable 
to. 


Especially  since  the  effects  of  non-uniform  albedo  and  intersection  are  buried  m  the  reflectance 
maps. 


i 
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Figure  2-6:  Introduced  global  coordinate  system. 

2.5  Simplifications 

The  equations  in  the  last  section  dealt  with  the  general  case  (subject  to  the  assump¬ 
tions  mentioned  in  that  section).  There  are  several  simplifications  that  make  the 
equations  easier  to  solve. 

2.5.1  Special  global  coordinate  system. 

So  far  all  the  equations  have  been  written  for  any  global  coordinate  system.  I  would 
now  like  to  restrict  the  equations  to  a  particular  global  coordinate  system,  namely  the 
coordinate  system  shown  in  Figure  2-6.  This  coordinate  system  is  defined  as  follows 

1.  Place  the  origin  of  the  global  coordinate  system  half  way  between  the  principal 
points  of  the  two  cameras.  (That  is,  place  the  origin  at  the  point  Pj  +  b/2.) 

2.  Choose  the  xo  direction  along  the  line  connecting  the  two  cameras, 

xo  =  b/||b)|.  (2.29) 


3.  Choose  Zo  as  the  average  optical  axis  direction  of  the  two  cameras  projected 
into  the  plane  perpendicular  to  Xo.  (That  is. 


(Zi  -  (2)  ■  Xq)Xq)  -f  (Z2  -  (Z2  ■  Xq)Xq) 
||(zi  -  (z,  •  Xo)Xo)  +  (z2  -  (z2  •  Xo)Xo)|| ' 


4.  Choose  y0  in  the  direction  of  z0  x  x<,  in  order  to  create  a  right  handed  coordinate 
system. 

5.  Also  set  up  a  virtual  image  plane  with  /  =  1. 

In  this  coordinate  system,  Pt  =  -b/2,  P2  =  b/2,  and  b  =  (6,0.0)r. 
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2.5.2  Removing  the  view  direction  dependence. 

A  common  simplification  for  computer  vision  is  the  assumption  of  a  Lambertian 
reflectance  map.  Since  a  Lambertian  surface  reflects  light  equally  in  all  directions, 
vve  see  from  Equation  2.7  that  the  radiance  function  is  not  dependent  on  the  viewing 
direction.  Thus  the  dependence  on  v  can  be  removed  from  all  the  equations. 

Note  that  the  viewing  direction  can  also  be  removed  from  the  equations  when  the 
field  of  view  is  small.  In  this  case,  the  viewing  direction  is  approximately  the  same 
for  all  points  on  the  surface  and  its  effect  can  be  subsumed  into  the  reflectance  map. 
Doing  this  would,  of  course,  introduce  an  error  into  the  calculations.  This  error  would 
be  small  for  photo-topography  since  the  cameras  are  so  far  away  from  the  surface. 
The  large  viewing  distance  requires  the  use  of  a  telephoto  lens  which  has  a  small  field 
of  view. 


2.5.3  Constant  albedo. 

Thus  far  the  equations  have  included  terms  that  denote  position  on  the  surface  £. 
The  main  reason  for  this  dependence  is  to  take  into  account  varying  albedo,  varying 
reflectance  properties,  or  both.  To  simplify  the  situation  we  could  assume  that  the 
reflectance  properties,  albedo,  or  both  are  constant  across  the  surface.  Assuming  the 
reflectance  properties,  but  not  the  albedo,  are  constant  across  the  surface  results  in 
a  reflectance  function  that  is  separated. 

#(£;n:v)  =  p(£)#(n;v)  (2.31) 

where  R(h:v)  is  the  reflectance  function  for  a  surface  with  uniform  albedo,  no  in¬ 
terflection.  and  no  self  occlusion.  As  for  R ,  any  light  source  effects  are  included  in 

R. 

When  both  the  reflectance  an<^  albedo  are  constant,  the  dependence  of  the  re¬ 
flectance  map  on  surface  position  can  be  removed. 


R(Z\n;  v)  =  R(n;ir).  (2.32) 

Combined  with  either  Lambertian  reflectance8  or  when  the  field  of  view  is  small,  the 
dependence  on  v  can  be  dropped  also. 

#(f:n;  v)  =  R(n).  (2.33) 

This  is  the  representation  of  the  reflectance  function  that  is  seen  most  often  in  the 
vision  literature.  The  simplification  restricts  the  applicability  of  that  research  so 


7This  is  true  for  any  reflectance  function  that  is  view  independent,  not  just  Lambertian 
reflectance 

*0r  any  other  view  independent  reflectance  function 
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Figure  2-7:  Coordinate  systems  with  aligned  optical  axes. 

that  only  uniformly  colored  surface  patches  have  the  possibility  of  being  estimated 
correctly.  When  algorithms  based  on  this  simplification  axe  applied  to  images  that 
violate  these  simplifications,  we  would  expect  errors  at  the  transition  between  different 
colored  parts  of  the  surface,  within  differently  colored  areas,  or  both.  Trying  to 
estimate  both  the  surface  topography  and  surface  albedo  substantially  increases  the 
number  of  unknown  variables  (possibly  by  a  factor  of  2)  and  significantly  slows  down 
the  convergence  (see  Section  8.1). 


2.5.4  Aligned  cameras. 

The  final  simplification  that  can  be  made  is  to  align  the  cameras  so  that  their  optical 
axes  are  parallel  ( which  will  also  be  parallel  to  the  global  coordinate  system's  z0  axis 
by  construction).  This  coordinate  system  is  shown  in  Figure  2-7.  When  the  cameras 
are  aligned,  the  rotational  transformations  T\  and  T2  are  identity  transformations 
which  simplifies  the  stereo  equations  (Equations  2.19)  considerably  to 


R-i  —  Z  —  Pi  i 
R?  =  S  ~  P  2 


(2.34) 


While  this  situation  is  very  unrealistic  for  the  photo-topography  problem,  any  set  of 
images  can  be  re-projected  into  this  coordinate  system  (see  Section  8.5).  Hence  this 
simplification  does  not  seriously  restrict  the  applicability  of  the  research. 


2.5.5  The  simplified  equations. 

The  rest  of  the  thesis  is  based  on  equations  that  take  into  account  all  of  the  simplifi¬ 
cations  described.  In  particular,  the  simplifications  in  the  following  list  are  made: 
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•  A  special  global  coordinate  system  that  is  halfway  between  the  two  camera 
positions  is  used. 

•  All  surface  points  are  assumed  to  be  visible  from  the  two  cameras. 

•  The  reflectance  properties  of  the  surface  are  assumed  to  be  constant  with  no 
interflection  and  no  mutual  occlusion. 


•  The  radiance  properties  of  the  surface  are  assumed  to  be  Lambertian,  allowing 
the  view  direction  dependence  to  be  dropped  from  the  reflectance  equations. 

•  The  surface  is  assumed  to  have  constant  albedo  allowing  the  position  depen¬ 
dence  to  be  dropped  from  the  equations. 

•  The  camera  optical  axes  are  assumed  to  be  aligned  with  each  other  allowing  the 
rotational  transforms  to  be  dropped  from  the  stereo  constraint  equations. 


Unless  noted,  the  rest  of  the  thesis  assumes  that  all  these  simplifications  hold.  I  will 
specifically  point  out  results  that  apply  to  the  more  general  case.  Taking  into  account 
all  of  these  simplifications  we  find  that  the  simplified  photo-topography  problem  is 
constrained  such  that 

£(1)(r,)  =  R(1)(  n) 

E<”M  =  ,2'33’ 


where 


(sc  +  W2 )/  _  £  +  b/2 

Fl  (£  +  b/2)  •  z,  f-z,  ’ 
(s  ~  b/2)/  _  £  —  b/2 
F2  (£  -  b/2)  •  z2  i-ii 


(2.36) 


If  we  define  c  =  £  •  z0  and  r  =  /£/-,  then  the  constraint  equations  can  be  written 


£(1,(r+^)  =  fl(1)(n). 


(2.37) 


The  definitions  for  z  and  r  above  introduces  a  peculiarity  into  the  equations:  the 
units  for  z  and  r  are  not  the  same.  In  particular,  r.  b.  and  £  are  in  planet  units  (say 
miles),  while  r  and  /  are  in  camera  units  (say  millimeters). 

I  find  it  convenient  in  subsequent  chapters  to  use  a  slightly  different  version  of 
these  last  equations  where  the  components  of  r  =  (x,  y ,  /)  are  explicit  and  the  normal 
vector  n  is  parameterized  using  gradient  components  p  and  q. 
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Figure  2-8:  Camera  calibration  geometry. 


where 


P  = 
<?  = 


fzt 


j.:z  +  z 

fzy 

y:v  +  z' 


(2.39) 


Equation  2.39  are  referred  to  as  the  Integrability  Constraint  Equations  since  they 
relate  the  surface  normal  components  to  the  partial  derivatives  of  the  surface  height 
c.  The  equations  can  be  used  constraint  the  values  of  p  and  q  that  are  consistent 
with  an  underlying  surface. 

Writing  Equations  2.37  in  terms  of  (r,  y )  and  p  and  q  produces  the  Photo-topography 
Constraint  Equations 

Eil)(x  +  ^,y)  =  R{1)(p.<i), 

fb  (2-40> 

E{2)(x  -  — ,y)  =  R{2)(p,q). 

I  will  be  working  with  these  equations  in  the  chapters  that  follow. 


2.6  Camera  Calibration 

In  order  to  relate  positions  in  the  image  to  direction  vectors  in  3-space,  the  origin 
of  the  camera  coordinate  system  must  be  known.  Finding  this  origin  is  part  of  the 
interior  orientation  problem  of  classical  optics.  As  mentioned  in  Section  2.2.4,  this 
origin  is  the  principal  point  in  the  image  plane.  In  the  special  coordinate  system 
of  Figure  2-8,  the  position  of  the  image  plane  origin  with  respect  to  each  camera 
coordinate  system  origin  can  be  specified  by  a  vector  v  =  (vx,  vv,  f)T.  These  vectors 
specify  the  offset  (in  pixel  coordinates)  of  the  image  plane  for  each  camera.  Suppose 
u0  is  the  pixel  position  of  an  object  point  in  the  global  image  and  v0  is  the  position 


A 
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of  the  origin  of  the  global  image  plane  in  global  coordinates,  then  given  the  pixel 
position  of  the  projection  of  this  this  same  object  point  in  the  camera  images  u;  and 
u2.  the  offset  of  each  image  plane  is 


(2.41) 

(2.42) 


The  values  of  vt  and  v2  can  be  quite  large  in  the  aligned  coordinate  system  indicatin 
that  the  images  must  be  shifted  far  away  from  the  camera  coordinate  system  origin. 
While  this  is  not  possible  physically,  it  is  a  consequence  of  re-projecting  real  images 
into  the  aligned  coordinate  system.9 10 


9This  is  especially  true  with  the  field  of  view  is  small 

10See  Section  8.5  for  how  to  do  this 
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Chapter  3 


Fusion  Strategy 


This  chapter  presents  my  general  solution  strategy  to  the  photo- topography  problem. 
The  idea  is  to  closely  couple  the  solution  of  shape-from-shading  and  stereo  so  that 
each  can  help  the  other.  The  important  thing  is  to  take  into  account  the  strengths  of 
each  method  and  use  each  method  to  cover  the  weaknesses  of  the  others. 

The  first  step  in  solving  the  photo- topography  problem  is  to  analyze  the  infor¬ 
mation  available  as  part  of  the  problem.  The  photo-topography  problem  has  two 
information  sources:  1)  the  gray  levels  in  the  each  image  are  an  indication  of  the 
surface  orientation  with  respect  to  the  light  source  and  2)  assuming  corresponding 
pixels  in  each  image  can  be  matched  up,  the  stereo  information  can  be  used  to  recover 
the  shape.  Of  the  two,  the  gray  level  information  is  more  directly  accessible  since 
no  correspondence  must  be  found.  Finding  the  correspondence  between  pixels  is  a 
classical  problem  for  stereo  algorithms  and  is  very  hard. 

The  photo-topography  images  also  offer  one  other  source  of  information.  Since 
the  images  are  typically  taken  with  two  different  light  source  positions,  the  gray 
levels  of  corresponding  pixels  constrain  the  set  of  possible  surface  orientations  as  they 
would  for  photometric  stereo  [YVoodham,  1980].  For  Lambertian  reflectance,  this  set 
contains  at  most  two  orientations.  Since  extracting  this  information  relies  on  pixel 
correspondence,  it  has  the  same  limitations  as  the  stereo  information. 

When  designing  a  fusion  algorithm  it  is  also  important  to  analyze  whether  the 
information  sources  are  independent  or  not.  By  independent  I  mean.  ‘Is  it  possible 
to  differentiate  between  the  two  information  sources'?  Or  said  another  way.  ‘Given  a 
combined  signal  containing  two  or  more  information  sources  is  it  possible  to  estimate 
the  relative  contributions  of  each  source'?  Or  said  a  third  way,  ‘Given  a  combined 
signal  are  the  information  components  observable  ? 

Doing  this  analysis  on  the  photo-topography  images  we  find  that  the  informa¬ 
tion  sources  are  indeed  independent.  The  shading  information  is  strongest  when  the 
shading  is  smooth,  while  the  stereo  information  is  strongest  near  surface  discontinu¬ 
ities  (which  contribute  to  shading  discontinuities)  and  when  the  cameras  are  widely 
separated.  The  photometric  stereo  information  is  strongest  when  the  light  source 
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Shading 

Stereo 

Lighting 

Shape  from  Shading 

Shape  from 
shading 

Surface 

constraint 

Surface 

orientation 

constraint 

Binocular  Stereo 

Correspondence 

constraint 

Binocular 

stereo 

Correspondence 

constraint 

Photometric  Stereo 

Correspondence 

constraint 

Cor.espondence 

constraint 

Photometric 

stereo 

Table  3-1:  Photo- topography  problem  information  sources 


positions  are  widely  separated. 

Given  the  previous  analysis  we  would  expect  that  the  best  estimates  of  the  surface 
would  be  found  when  the  images  contain  large  regions  of  smooth  gray  level  changes 
intermixed  with  areas  of  rapidly  changing  gray  level,  the  cameras  are  well  separated, 
and  the  light  source  positions  are  separated. 

There  are  several  possible  fusion  paradigms  as  I  discussed  in  Chapter  1.  The 
most  promising  fusion  paradigms  take  into  account  the  most  amount  of  information. 
The  modularized  approaches  (where  existing  vision  algorithms  are  applied  to  the 
problem  individually  and  where  these  individual  estimates  are  then  combined  after¬ 
wards)  exploit  some  of  the  information  but  not  all  of  it.  In  particular,  if  you  think 
of  these  problems  in  terms  of  a  table  or  matrix  as  in  Table  3-1,  the  modularized 
approaches  exploit  the  information  along  the  diagonal  but  do  not  take  into  account 
the  off-diagonal  coupling  between  methods.  On  the  other  hand,  I  use  a  close  coupled 
approach  based  on  variational  calculus  that  exploits  all  the  coupling  inherent  in  the 
problem.  This  approach  is  not  a  panacea,  however,  as  we  will  see  later.  It  is  still 
possible  to  choose  cost  functions  that  don’t  encourage  cooperation  among  the  var¬ 
ious  information  sources.  However,  by  understanding  the  couplings  inherent  in  the 
problem  a  suitable  cost  function  can  usually  be  found. 


3.1  Variable  representations 

The  most  important  decision  when  developing  a  fusion  algorithm  is  the  variable 
representation.  Choose  the  right  representation  and  everything  will  work  together 
smoothly.  Choose  the  wrong  representation  and  the  algorithm  may  be  hindered  by 
slow  convergence,  local  minima,  discontinuities,  etc.  Aside  from  this  performance 
effect,  how  can  a  good  representation  by  recognized?  I  can’t  offer  a  general  solution, 
but  I  can  offer  some  rules  of  thumb. 

•  A  good  representation  captures  all  the  necessary  information  in  the  problem 
with  no  redundancy.  For  example  it  is  better  to  use  a  parameterized  family  of 
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functions  rather  than  a  digitized  version  of  the  same  function  if  the  number  of 
parameters  would  be  much  less  than  the  number  of  sampled  points. 

•  A  good  representation  can  easily  capture  the  constraints  of  the  problem.  For 
example,  it  is  better  to  use  a  function  that  automatically  matches  the  boundary 
conditions  of  a  problem  rather  than  to  impose  the  constraint  using  penaltv 
functions  or  Lagrange  multipliers. 

•  A  good  representation  usually  has  equations  that  are  simple  and  easy  to  under¬ 
stand. 

•  A  good  representation  is  as  close  as  possible  to  the  problem.  It  is  not  removed 
by  several  integrations,  differentiations,  or  other  mappings  that  can  introduce 
non-linearities,  bias,  drift,  or  additional  unobservable  parameters. 

•  A  good  representation  allows  for  easy  information  exchange  between  the  differ¬ 
ent  information  sources  in  the  problem.  Such  a  representation  would  directly 
express  constraints  from  one  information  source  so  that  the  other  sources  will 
automatically  take  them  into  account. 

The  variable  representation  defines  the  space  that  will  be  searched.  Usually,  so¬ 
lution  spaces  that  are  lower  dimensional,  smoother,  and  more  bowl-like  will  make 
finding  the  solution  faster  and  easier. 

I  present  four  different  variable  representations  for  the  photo-topography  problem 
in  Chapter  4.  The  four  representations  highlight  the  importance  of  choosing  the  right 
representation. 


3.2  Cost  functions 

After  the  choice  of  variable  representation,  the  choice  of  cost  function  is  the  next 
important.  Like  choosing  the  right  variable  representation,  choosing  the  right  cost 
function  will  affect  the  performance  of  the  algorithm  and  the  algorithm's  robustness. 

Building  a  closely  coupled  fusion  algorithm,  requires  that  all  the  Governing  equa¬ 
tions.  constraints,  and  desired  outcomes  be  formulated  into  a  single  cost  function, 
usually  some  sort  of  energy-based  functional.  Since  everything  is  combined  together 
to  generate  a  single  scalar  value  (the  cost),  the  algorithm  is  free  to  perform  trade-offs. 
For  instance,  the  algorithm  can  trade-off  accuracy  of  the  result  in  order  to  satisfy 
another  constraint  (such  as  smoothness).  The  types  of  trade-offs  that  are  allowed 
and/or  favored  can  be  controlled  by  changing  the  relative  weights  between  terms  in 
the  cost  function,  or  by  totally  reformulating  the  cost  function  (1  show  examples  of 
both  in  Chapter  4). 

Cost  functions  are  relatively  easy  to  formulate  once  a  variable  representation  has 
been  chosen.  Simply  write  the  governing  equations  in  terms  of  these  variables,  and 
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create  the  cost  function  directly  from  these  equations.  If  necessary,  sum.  integrate  or 
somehow  combine  the  results  from  the  constraints  to  form  a  scalar  cost.  It  might  be 
necessary  to  add  some  regularizing  terms  to  the  cost  function  in  order  to  produce  a 
well  posed  problem  (see  [Tikhonov  and  Arsenin.  1977]  or  [Hadamard.  1923]). 

Many  vision  problems  require  some  type  of  regularization  to  constrain  the  solut  ion. 
Typically,  smoothness  is  assumed  for  some  variable  (such  as  surface  height)  and  is  used 
to  create  a  smoothness  regularization  term.  Regularization  terms  are  required  when 
there  are  too  many  solutions  to  a  given  problem.  The  regularization  terms  can  then 
be  used  to  constrain  the  set  of  possible  solutions  to  be  physically  reasonable.  In  all  the 
photo-topographv  algorithms  presented  later  in  this  chapter  I  have  included  a  surface 
smoothness  term.  While  the  smoothness  term  is  not  strictly  required  (since  shape- 
from-shading  problems  do  have  unique  solutions),1  the  regularization  term  serves  to 
speed  up  convergence  by  making  the  solution  space  more  bowl-like. 

Constraints  can  be  added  to  a  candidate  cost  function  using  the  method  of  La¬ 
grange  multipliers  or  by  adding  a  penalty  function.  The  Lagrange  multiplier  method 
is  used  for  hard  constraints,  while  the  penalty  functions  are  used  for  soft  constraints. 
Hard  constraints  are  required  to  be  met  exactly  by  the  solution,  while  soft  constraints 
need  not  be  met  exactly.  Hard  constraints  restrict  the  space  of  the  feasible  solutions, 
while  soft  constraints  create  bowl-litce  edges  on  the  solution  space.  Both  methods  can 
be  used  to  help  convergence  to  a  solution  and  to  restrict  the  search  space  to  more 
desirable  solutions. 

Different  cost  functions  can  be  generated  from  the  same  set  of  equations  and 
variable  representation  by  changing  the  desired  outcome.  For  instance,  the  cost  func¬ 
tion  for  determining  the  surface  slopes  from  the  photo-topographv  problem  would  be 
different  from  the  cost  function  for  finding  the  surface  height. 


3.3  Solution  Techniques 

A  typical  cost  function  for  a  vision  problem  is  of  the  form 

min«/  =  JJ  L(w.  u'\  xi"\ . . .)  dx  dy  (3.1) 

where  u  are  the  optimization  variables,  and  L  is  a  possibly  non-linear  function  of 
the  optimization  variables  and  its  derivatives.  The  integral  is  taken  over  the  domain 
of  the  optimization  variables  or  the  problem.  The  solution  to  this  problem  can  be 
found  by  solving  the  associated  Euler- Lagrange  equations  (see  a  variational  calculus 
book  such  as  [Courant  and  Hilbert,  1962]  or  Horn's  Appendix  [Horn.  1986]  for  more 
details). 

The  Euler- Lagrange  equations  for  a  problem  such  as  the  one  above  are  typically 
coupled  non-linear  equations.  Such  equations  are  usually  very  difficult  to  solve  ana- 

'When  the  images  contain  singular  points  [Saxberg,  1989] 
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lyticallv  but  can  sometimes  be  solved  numerically  by  converting  them  into  discrete 
equations.  The  conversion  process  involves  substituting  discrete  approximations  for 
any  derivatives  of  the  optimization  variables.  The  optimization  variables  may  have 
to  be  approximated  bv  a  discrete  vector  as  well.  The  equations  are  then  re-arranged 
to  create  iterative  update  equations  of  the  form 

u[k  +  1)  =  f(u{k).u(k  -  1),. . .)  (3.2) 

where  u(k)  is  the  value  of  the  optimization  variables  for  the  A--th  iteration. 

When  u  has  many  components  and  when  the  components  are  updated  in  sequence 
based  on  the  most  current  estimate  u,  the  resulting  update  scheme  is  called  a  Gauss- 
Setdel  optimization.  When  all  of  the  components  of  u  are  updated  simultaneouslv 
based  on  a  previous  estimate  for  u.  the  resulting  scheme  is  called  a  Gauss-Jordan 
optimization.  Gauss-Seidel  optimization  schemes  have  higher  convergence  rates  and 
are  more  robust,  and  are  best  implemented  on  a  serial  computer.  Gauss-Jordan 
schemes,  while  they  have  lower  convergence  rates  and  are  not  as  robust,  can  be 
implemented  on  parallel  computers.  Parallel  computations  can  produce  results  faster. 

Another  way  of  solving  the  optimization  problem  posed  above  is  by  using  direct 
optimization  techniques.  In  this  case  the  cost  function,  instead  of  the  Euler- Lagrange 
equations,  is  discretized.  Any  integrations  are  approximated  by  sums  and  any  deriva¬ 
tives  are  approximated  by  differences.  The  resulting  cost  function  is  of  the  form 

min  J  =  /(«)  (3.3) 

I  V 

where  f(u)  is  a  discrete  approximation  of  L(u:  u';  u': . . .).  Any  of  the  wide  range  of 
optimization  algorithms  that  are  in  the  literature  can  then  be  applied  to  this  problem. 
For  vision  problems,  conjugate  gradient  optimization  shows  the  most  promise.  The 
conjugate  gradient  scheme  doesn’t  require  the  formation  of  the  problem  Hessian  (a 
linear  approximation  to  the  second  order  derivative  of  the  solution  space  at  a  given 
point),  which  for  an  optimization  problem  with  N  variables  is  an  .V-by-.V  matrix.  The 
conjugate  gradient  scheme  is  important  for  vision  problems  since  for  a  typical  256-by- 
256  image,  the  shape-from-shading  problem  would  have  2562  or  65536  optimization 
variables.  The  Hessian  for  this  problem  would  have  2564  or  over  4  billion  elements! 

Conjugate  gradient  optimization  requires  that  both  the  cost  function  and  its  gra¬ 
dient  are  computable.  While  an  approximation  to  the  gradient  of  the  cost  function 
can  be  computed  using  finite  differences,  this  approach  is  usually  slow  (since  it  re¬ 
quires  at  least  A'  function  evaluations  for  each  gradient  evaluation).  I  use  analytically 
determined  gradients  in  all  the  algorithms  presented  later  in  this  chapter.2 

One  strategy  that  can  also  be  used  to  solve  non-linear  problems  is  the  method  of 
homotopy  or  the  continuation  method.  The  continuation  method  involves  solving  a 

2 Verified  by  comparing  with  numerical  gradients 
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series  of  related  problems  of  increasing  difficulty  that  have  similar  solutions.  Typically, 
the  original  problem  is  restated  to  include  a  parameter,  such  as  3.  which  controls  the 
difficulty  of  the  problem.  When  3=1  (for  instance),  the  problem  is  easy  to  solve 
or  has  a  analytical  solution.  When  3  =  0  (for  instance)  the  original  problem  is 
recovered.  To  solve  the  original  problem,  a  series  of  problems  are  solved  while  3 
is  slowly  decreased.  The  solution  of  each  subsequent  problem  is  used  as  the  initial 
condition  for  the  next  problem. 

One  simple  way  to  use  the  continuation  technique  is  to  introduce  a  regularization 
term  based  on  3  that  creates  a  convex  problem  when  3=1, 

+  0s(u)-  (3-4) 

*  y 

where  g{u)  is  a  regularization  function  that  forces  the  problem  to  be  convex.3  During 
optimization,  3  is  slowly  reduced  to  zero,  where  the  original  problem  is  recovered. 
The  hope  with  the  continuation  method  is  that  the  solutions  to  the  series  of  problems 
will  be  close  to  the  solution  of  the  original  problem  so  that  they  are  good  initial 
conditions. 

I  use  a  form  of  the  continuation  based  method  for  the  algorithms  in  this  thesis. 
Since  shape-from-shading  problems  have  a  well  defined  solution  when  singular  points 
are  in  the  image,  the  smoothness  terms  axe  not  needed  to  guarantee  a  solution. 
The  smoothness  terms  do.  however,  help  convergence.  During  optimization.  I  slowly 
reduce  the  smoothness  parameter  toward  zero  in  order  to  avoid  biasing  the  solution 
[Horn  and  Brooks,  1986]. 

The  questions  of  existence  and  uniqueness  come  up  when  working  with  optimiza¬ 
tion  algorithms  such  as  those  presented  in  this  thesis.  For  the  types  of  cost  functions 
presented  in  this  thesis,  it  is  clear  that  a  solution  exists;  the  cost  functions  are  bounded 
from  below  by  zero.  That  is,  the  best  possible  value  for  the  cost  function  is  zero  and 
can  be  achieved  only  when  the  estimated  surface  images  and  the  actual  images  match 
exactly  and  when  any  regularization  terms  are  set  to  zero. 

The  uniqueness  of  a  solution  depends  a  great  deal  on  the  surface  to  be  estimated. 
In  general,  both  global  and  local  minima  will  exist  (as  evidenced  by  the  Hard  Crater 
problem  discussed  in  Chapter  7).  The  optimization  techniques  discussed  above  only 
guarantee  convergence  to  a  local  minima.  The  global  minimum  may  only  be  achieved 
if  the  initial  conditions  for  the  optimization  algorithm  are  close  to  the  true  solution. 


3.4  Speed-up  Techniques 

A  very  well  researched  part  of  optimization  theory  is  how  to  speed  up  the  conver¬ 
gence.  For  vision  problems  there  are  two  promising  speed  up  technologies:  the  use 

3Convex  problems  have  the  property  that  the  solution  space  is  essentially  bowl-like,  with  no  local 
minima  and  a  single  global  minima  which  makes  them  very  easy  to  solve. 


(Equations  2.19). 
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Figure  3-1:  Hierarchical  basis  functions.  Shown  are  the  hierarchical  basis  nodes  at  each  level 
(circles,  triangles,  etc.)  and  the  associated  interpolation  function  extent  for  the  front-most 
nodes.  The  nodal  basis  for  the  9- by- 9  domain  would  have  nodes  at  each  pixel  location  and 
an  extent  half-again  as  small  as  the  level  1  extents  shown. 

of  hierarchical  basts  functions ,  and  multi-grid  methods.  Both  methods  try  to  speed 
up  the  optimization  problems  by  increasing  the  information  transfer  spatially.  The 
methods  are  based  on  the  property  of  many  vision  algorithms  that  an  optimization 
variable  within  a  grid  of  optimization  variables  may  only  be  affected  by  its  nearest 
neighbors.  Due  to  the  local  connectedness,  many  vision  algorithms  have  diffusion- 
like  properties;  the  solution  must  diffuse  throughout  the  grid.  Schemes  that  directly 
transfer  information  over  longer  distances  instead  thus  may  speed  up  an  algorithm. 

Using  hierarchical  basis  functions  transforms  the  optimization  space  as  seen  by 
the  optimization  algorithm  but  not  as  seen  by  the  vision  algorithm.  Basically  it's  like 
a  change  of  basis.  Figure  3-1  shows  how  a  9-bv-9  domain  would  be  represented  in  hi¬ 
erarchical  basis.  In  particular,  note  that  the  nodes  of  the  hierarchical  basis  propagate 
information  over  a  much  larger  range  (due  to  their  extended  interpolation  extent) 
than  the  nodes  in  the  nodal  basis  (particularly  for  the  nodes  at  the  upper  levels). 
The  figure  shows  linear  interpolation  between  nodes,  but  any  interpolation  scheme 
can  be  used  to  build  a  hierarchical  basis  (see  [Szeliski.  1990]).  The  problem  is  solved 
as  before  but  with  the  new  variables.  If  necessary,  the  variables  can  be  transformed 
to  the  nodal  basis  when  computing  the  cost  function  or  gradient.  Unfortunately, 
all  these  transformations  have  the  potential  to  introduce  round-off  errors  which  can 
adversely  affect  sensitive  algorithms. 

The  hierarchical  basis  functions  have  the  most  effect  on  the  convergence  and  are 
the  easiest  to  implement  when  the  grid  size  is  2"  +  1  where  n  is  any  positive  in¬ 
teger.  For  such  a  grid  it  is  possible  to  use  n  +  1  hierarchical  basis  levels.  Using 
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Figure  3-2:  Non-zero  Hessian  elements  for  a  9-by-9  image.  The  plot  on  the  left  shows  the 
non-zero  Hessian  elements  when  the  nodal  basis  is  used.  The  plot  on  the  right  shows  the 
non-zero  elements  when  a  full  hierarchical  basis  is  used. 


hierarchical  basis  functions  increases  the  communication  between  nodes  in  the  image 
array  as  shown  in  Figure  3-2.  This  increased  communication  speeds  up  the  diffu¬ 
sion  process  considerably.  I  have  noticed  a  five-fold  increase  in  convergence  for  the 
photo-topography  algorithms. 

The  multi-grid  methods  seek  to  propagate  information  over  a  larger  range  by 
solving  a  series  of  problems  of  different  size.  Usually  the  original  problem  is  formulated 
on  grids  that  decrease  in  size  by  a  factor  of  two  when  going  from  one  layer  to  the 
next  (see  Figure  3-3).  The  solutions  on  one  layer  are  related  to  solutions  on  the  layers 
above  and  below  via  interpolation  or  prolongation.  The  solutions  are  kept  consistent 
with  each  other  via  both  intra- layer  and  inter-layer  processes  (see  [Terzopoulos,  1984] 
and  [Brandt  and  Dinar,  1979]). 

Multigrid  m  *thods  have  the  potential  to  be  much  faster  than  the  hierarchical  basis 
functions  since  most  of  the  computation  (and  optimization)  is  done  on  the  smaller 
layers.  Multigrid  methods  are  well  suited  to  linear  problems  (such  as  surface  fitting), 
but  may  not  work  for  non-linear  problems.  The  problem  is  that  typically 

H9(«)^ffCCu)  (3.5) 


for  non-linear  functions  g( u),  and  the  multi-grid  methods  require 
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Figure  3-3:  Multigrid  levels 

to  constrain  the  solutions  on  the  smaller  grids  so  that  they  don't  bias  the  solution  on 
the  larger  grid. 

One  type  of  multi-grid  method  that  can  be  used  for  non-linear  problems  is  the 
coarse-to-fine  method.  In  this  method,  the  problem  is  solved  on  coarse  layers  first 
and  the  solution  to  each  layer  provides  the  initial  condition  to  the  next  finer  layer 
below.  This  method  is  significantly  faster  than  just  optimizing  on  the  finest  grid  but 
doesn’t  produce  as  much  convergence  speed-up  as  the  full  multi-grid  method. 

Like  the  hierarchical  basis  methods,  the  multi-grid  methods  work  best  when  the 
grid  size  is  2"  +  1.  However,  since  the  multi-grid  methods  define  a  series  of  problems 
rather  than  just  choosing  a  new  set  of  basis  functions,  the  multi-grid  methods  can  be 
implemented  easily  for  all  grid  sizes.  There  is  some  evidence,  though,  that  grid  size 
reductions  should  be  near  2  for  best  convergence  rates  [Terzopoulos.  1984]. 
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Chapter  4 

Candidate  algorithms 


In  this  chapter  I  discuss  several  candidate  algorithms  for  the  photo-topographv  prob¬ 
lem.  Since  each  of  the  algorithms  estimate  the  surface  depth  using  a  combination  of 
shape-from-shading  and  stereo  I  call  them  Depth  From  Shading  and  Stereo  (DFSS) 
algorithms.  Along  the  way  I  will  discuss  the  rationale  behind  each  algorithm,  its 
strengths  and  weaknesses,  and  how  it  fits  in  with  the  fusion  techniques  discussed  so 
far.  The  performance  of  these  algorithms  on  a  set  of  synthetic  test  images  is  presented 
in  Chapter  5.  The  performance  of  the  z-only  algorithm  on  real  images  is  presented 
in  Chapter  9. 

The  approach  I  am  taking  to  fuse  the  vision  algorithms  is  to  develop  a  single  cost 
function  that  incorporates  the  problem  constraints  along  with  some  regularization 
terms  to  help  direct  the  search  path.  This  approach  differs  from  many  researchers' 
attempts  at  fusion  algorithms  in  that  it  closely  integrates  the  methods  rather  than 
building  a  module-based  solution.  I  believe  that  this  approach  will  create  more  robust 
solution  methods. 

In  effect,  I  pose  a  generalized  optimization  problem  which  can  be  solved  many 
ways.  I  have  tried  various  solution  methods  with  this  research  and  have  chosen 
direct  optimization  via  the  conjugate  gradient  method  as  my  preferred  method.  The 
conjugate  gradient  method,  like  all  direct  optimization  methods,  guarantees  reduction 
of  the  cost  function  at  each  step,  in  contrast  to  the  more  widely  used  Gauss- Seidel 
or  Gauss-Jordan  methods.  The  big  advantage  of  the  conjugate  gradient  method  over 
other  direct  optimization  methods  is  that  no  Hessian  needs  to  be  computed  or  stored. 
Computer  vision  algorithms  have  thousands  of  free  variables  which  would  result  in  a 
Hessian  with  millions  of  terms  if  it  were  computed. 

I  have  looked  at  four  basic  algorithms  for  the  photo-topography  problem. 

•  The  zpq  and  z-only  algorithms  estimate  everything  in  a  single  global  coordi¬ 
nate  system  that  is  defined  to  be  halfway  between  the  two  camera  positions.  I 
refer  to  these  algorithms  as  centralized.  Figure  4-1  shows  the  flow  of  a  typical 
centralized  algorithm  as  a  tree  diagram.  The  current  estimate  of  the  surface 
height  z  is  used  to  project  points  in  the  global  coordinate  system  to  points  in 
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Figure  4-1:  Centralized  algorithm  tree. 
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Figure  4-2:  Decentralized  algorithm  tree. 

each  image  (via  perspective  projection).  These  points  won’t  in  general  land 
on  a  pixel  center  so  some  type  of  interpolation  (e.g.,  bilinear  or  bicubic  inter¬ 
polation)  is  used  to  determine  the  value  of  the  image  at  the  projected  points. 
This  interpolated  image  F  is  then  compared  to  a  computed  image  based  on  the 
current  estimate  of  the  surface.  The  error  is  used  to  update  p,  q,  and  ultimately 
z.  This  type  of  algorithm  closely  integrates  the  constraints  from  both  shading 
and  stereo. 

•  The  dual-z  algorithm  determines  estimates  of  the  surface  for  each  image  sepa¬ 
rately  and  takes  into  account  the  stereo  information  via  a  penalty  term.  I  refer 
to  this  algorithm  as  decentralized.  Figure  4-2  shows  the  flow  of  this  type  of 
algorithm.  The  estimated  height  for  each  image  is  used  to  compute  the  cor¬ 
responding  image  estimate.  The  errors  between  this  estimated  image  and  the 
actual  image  for  each  camera  are  then  used  to  update  the  associated  p,  q ,  and  z 
for  each  image.  So  far  this  is  the  same  as  for  a  normal  shape-from-shading  algo¬ 
rithm.  However,  instead  of  just  updating  the  surface  height  estimates  directly 
from  the  image  errors,  the  algorithm  also  takes  into  account  the  differences  be¬ 
tween  the  two  surface  estimates.  This  type  of  algorithm  doesn’t  integrate  the 
shading  and  stereo  information  as  closely  as  the  centralized  algorithms. 

•  The  disparity  algorithm  is  also  centralized  algorithm  but  uses  estimates  of  the 
disparity  instead  of  the  surface  depth  as  the  fundamental  variable.  Figure  4-3 
shows  the  flow  for  this  algorithm.  The  current  estimate  of  the  disparity  u  is  used 
to  project  points  in  the  global  coordinate  system  to  points  in  each  image  (via 
perspective  projection).  These  points  won’t  in  general  land  on  a  pixel  center 
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Figure  4-3:  Centralized  algorithm  tree  based  on  disparity. 

so  interpolation  is  used  to  determine  the  brightness  value  in  the  image  at  the 
projected  points.  This  interpolated  image  F  is  then  compared  to  a  computed 
image  based  on  the  current  estimate  of  the  surface.  The  error  is  used  to  update 
p,  q,  and  ultimately  u.  The  surface  height,  z,  is  computed  from  u  if  needed. 
This  type  of  algorithm  also  closely  integrates  the  constraints  from  both  shading 
and  stereo. 

The  algorithms  differ  mostly  in  the  representation  of  the  surface  topography.  The 
representation  chosen  for  the  optimization  variables  is  very  important  to  the  per¬ 
formance  of  computer  vision  algorithms.  If  the  right  representation  is  chosen  then 
the  search  space  will  be  smoother  and  contain  less  local  minima  than  if  the  wrong 
representation  is  chosen. 

4.1  zpq  Algorithm. 

This  algorithm  is  based  loosely  on  the  Height  and  Gradient  from  Shading  algorithm 
of  Horn  [Horn,  1989].  Following  Horn's  algorithm,  surface  height  z,  and  the  surface 
gradients  p,  and  q ,  are  used  as  optimization  variables.  The  two  photo-topography 
images,  camera  geometry,  and  surface  reflectance  functions  are  given  as  inputs  to 
the  algorithm.  The  equations  that  govern  this  situation  are  the  photo-topography 
constraint  equations  (2.40)  and  the  integrability  constraint  equations  (2.39).  The 
integrability  constraint  equations  are  necessary  to  ensure  that  p  and  q  are  consistent 
with  the  underlying  surface  z.  The  cost  function  for  this  algorithm  is  formed  by 
integrating  the  squared  error  introduced  by  the  current  estimates  for  p,  q ,  and  r. 
together  with  a  penalty  functions  for  departure  from  integrability  and  departure  from 
smoothness, 

min  J  =  ~  JJ +  ^,y)  -  +  (^E{2)(x  -  ~,y)  -  /?(2)(p,q)j 
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L  J 

+  ^  [pl  +  pi  S- ql  +  ql}}  dx  dy.  (4.1) 

This  cost  function  allows  the  normal  vector  components  p  and  q  freedom  to  try  to 
match  the  input  images  without  requiring  integrability.  The  penalty  functions  are 
then  used  to  bias  p  and  q  toward  solutions  that  are  integrable  and  smooth.  Using  soft 
constraints  like  those  above  provides  for  more  degrees  of  freedom  which,  it  is  hoped, 
will  result  in  fewer  local  minima.  The  disadvantages  of  these  extra  degrees  of  freedom 
are  a  non-exact  solution  (since  z  will  not  be  exactly  consistent  with  p  and  q).  and 
slower  convergence. 

The  cost  function  above  is  continuous  and  must  be  discretized  before  it  can  be 
optimized  (solved).  The  discretization  process  is  an  approximation  process.  The  idea 
is  to  approximate  the  integral  above  by  some  function  of  a  finite  (usually  small)  set  of 
variables.  Researchers  typically  use  either  a  series  representation  of  the  integrand  or 
a  sampled  version  of  the  integrand.  Examples  of  the  first  representation  are  Fourier 
series  and  finite  element  methods.  Examples  of  the  second  representation  are  finite 
difference  methods. 

I  have  used  finite  difference  methods  exclusively  for  the  research  presented  in  this 
thesis.  The  main  reason  for  this  is  that  the  images  (£<')(*,  y)  above)  are  provided  in 
digital  form.  In  these  images,  each  pixel  represents,  in  some  sense,  a  weighted  average 
of  the  brightness  falling  within  the  sensitive  area  of  the  corresponding  photosensor. 
Thus  each  image  is  actually  an  array  of  brightness  values.  Given  this  fact  it  makes 
sense  to  approximate  the  values  for  p,  q ,  and  z  as  arrays  of  gradient  components  or 
surface  depth,  as  the  case  may  be.  Choosing  this  digital  representation  results  in  a 
problem  of  the  finite  difference  class. 

Suppose  p,  q ,  and  z  are  represented  as  arrays,  as  discussed  above.  Suppose  also 
that  the  continuous  derivatives  of  the  underlying  functions  (such  as  z)  are  approxi¬ 
mated  by  finite  differences,  then  the  cost  function  can  be  approximated  by  the  discrete 
sum, 


min  J  = 


V  xzx  +  z)  \  yzy  +  zj 

Pi  +  Pi  +  <ll  +  f  j]  } 


+  M 
+  A 


(4.2) 


where  V  is  the  discrete  domain  of  the  underlying  variables  in  the  global  coordinate 
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Figure  4-4:  Relative  sizes  of  the  p,  q ,  and  z  arrays  in  comparison  to  the  image  arrays  for 
the  co-grid  representation. 


system.  M  and  N  are  the  row  and  column  dimensions  of  the  discrete  domain,  and  e  is 
the  grid  spacing  (assumed  to  the  same  in  both  the  x  and  y  directions).  The  F(,)(x,  y) 
are  interpolated  from  the  input  images  EM{x,y)  using  linear  interpolation, 

F(,)(x ±  g, y)  =  £«(*, y)  +  (x±£-i)  \E^(x  + 1,  jr)  -  £(,)(x, jr)]  (4.3) 

where 

x  =  floor(x  ±  — -). 

The  floor(x)  function  returns  the  greatest  integer  that  is  smaller  than  x 
Equation  4.3  assumes  that  x  is  sampled  on  a  unity-spaced  grid. 

4.1.1  Co-grid  Implementation 

Two  approaches  have  been  taken  to  implement  this  cost  function.  In  the  first  ap¬ 
proach,  the  p  and  q  arrays  are  the  same  size  as  the  image  arrays  and  the  ;  array  is 
one  pixel  larger  in  both  the  column  and  row  directions.  That  is,  the  p  and  q  functions 
are  sampled  on  the  co-grid  of  the  function  z  (see  Figure  4-4).  This  approach  uses 
face-centered  surface  derivatives  that  are  valid  in  the  center  of  each  2-by-‘2  “face" 
formed  by  the  z  grid.  In  this  approach,  the  x-  and  y-derivatives  are  approximated  by 


(4-4) 
.  Note  that 
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computational  molecules  of  the  form, 


(4.5) 


(4.6) 


(4.7) 


which  have  low  approximation  error.  The  diamond  operator  o  is  similar  to  (but 
different  from)  two  dimensional  convolution.1  In  addition,  this  approach  requires 
that  z  be  sampled  on  the  co-grid  when  computing  the  terms  in  the  cost  function  that 
contain  z.  Based  on  a  bilinear  approximation,  z  can  be  sampled  using 


z  — 


1 

4 


o  z. 


(4.9) 


The  resulting  algorithm  performs  as  well  as  the  algorithm  to  be  described  next  but 
has  the  disadvantage  that  the  hierarchical  basis  functions  cannot  be  used  to  full 
advantage  since  not  all  the  p.  q,  and  s  arrays  can  be  of  the  size  2n  +  1.  Recall  that 
the  hierarchical  basis  functions  are  easiest  to  implement  and  provide  for  optimum 
communication  between  pixels  in  the  array  when  the  row  and  column  dimensions  are 
of  size  2"  +  1  for  some  n.  In  addition,  multi-grid  methods  also  cannot  be  used  when 
the  variable  arrays  are  of  different  sizes. 

4.1.2  Matched-grid  Implementation 

In  the  second  approach,  p,  q,  and  z  are  chosen  to  all  be  the  same  size  as  the  image 
arrays.  In  such  a  representation,  all  the  functions  are  sampled  on  the  same  grid  (see 
Figure  4-5)  which  is  why  this  approach  is  called  the  matched-grid  representation.  This 
approach  uses  vertex-centered  surface  derivatives  that  are  valid  at  each  vertex  of  the 


‘Suppose  h( i,j)  is  s  computational  molecule  (as  an  array),  then  the  diamond  operation  is  defined 
as 

(hoz)(i,j)  =  ^Tz(i-  M  +k,j  -  N  +  m)h{k,m)  (4.8) 

k.m 

where  M  is  the  row  dimension  of  h  and  N  is  the  row  dimension  of  h.  The  definition  differs  from 
two-dimensional  convolution  in  that  h  is  not  “flipped” . 


4.1.  ZPQ  ALGORITHM. 


Figure  4-5:  Relative  sizes  of  the  p ,  q,  and  z  arrays  in  comparison  to  the  image  arrays  for 
the  matched-grid  representation. 

z  grid.  Computational  molecules  that  exemplify  this  approximation  are. 

□ 


which  have  higher  approximation  error  than  their  face-centered  counterparts.  In 
practice,  the  algorithm  based  on  this  approach  has  nearly  the  same  performance  as 
the  algorithm  based  on  the  face-centered  approach.  Since  p  and  q  are  the  same  size 
as  z,  some  type  of  approximation  must  be  made  at  the  array  edges  (boundaries). 
I  have  chosen  to  extrapolate  the  estimates  using  a  bicubic  interpolant  and  use  this 
extrapolated  version  of  each  estimate  in  subsequent  calculations  (see  Section  A.2). 
Using  the  extrapolated  estimate  is  similar  to  using  natural  boundary  conditions  in 
variational  calculus  methods,  and  allows  for  easy  computation  of  the  gradient. 

The  performance  of  the  matched-grid  zpq  algorithm  on  four  test  image  sets  is 
presented  in  Chapter  5.  This  algorithm  gets  stuck  in  a  local  minimum  on  the  hard 
crater  images  (as  do  ail  of  the  algorithms  to  be  presented),  but  does  reasonable  well 
on  the  other  test  images  (except  for  the  mountain  images).  As  a  whole,  this  algorithm 
produces  surfaces  that  are  too  smooth  and  takes  longer  to  converge  than  the  other 
algorithms. 


•  •  •  •  • 


•  •  •  • 
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Like  all  the  algorithms  to  be  presented,  this  algorithm  is  run  using  a  exponentially 
changing  smoothness  parameter  A.  and  integrability  parameter  p  (if  it  exists).  A 
large  initial  smoothness  parameter  helps  to  make  the  search  space  more  bowl-like 
at  the  expense  of  meeting  the  photo-topography  constraints.  When  the  smoothness 
parameter  is  small,  the  photo-topography  constraints  are  met  at  the  expense  of  a 
possibly  bumpy  surface.  A  small  initial  integrability  parameter  allows  the  surface 
gradient  components  p  and  q  to  wander  far  from  integrability  in  order  to  create  image 
estimates  that  match  the  input  images.  A  large  integrability  parameter,  requires  that 
these  image  estimates  be  created  so  that  p  and  q  are  nearly  integrable. 

While  it  is  not  required  that  the  optimization  parameters  be  ramped  in  this  fash¬ 
ion,  I  have  found  that  doing  so  allows  for  faster  convergence  over  optimizing  with  the 
parameters  fixed  at  their  final  values. 


4.2  c-only  Algorithm 

This  next  cost  function  is  similar  to  the  zpq  cost  function  except  that  it  uses  hard 
integrability  constraints  instead  of  soft  constraints.  With  hard  constraints  we  are 
guaranteed  that  any  solutions  obtained  will  be  feasible.  The  trade-off  is  that  the 
algorithm  will  have  less  degrees  of  freedom  to  work  with  and  may  be  more  susceptible 
to  local  minima. 

For  this  cost  function,  the  depth  map  z  is  the  only  optimization  variable  thus  the 
resulting  algorithm  is  called  the  z-only  algorithm.  The  surface  gradient  components 
p  and  q  are  computed  directly  from  the  depth  map.  As  before,  the  photo-topography 
images,  camera  geometry,  and  surface  reflectance  functions  are  inputs  to  the  algo¬ 
rithm.  The  constraint  equations  again  come  from  the  photo-topography  constraints 
and  integrability  constraints.  The  cost  function  is  formed  by  integrating  the  squared 
photo- topography  error  introduced  by  the  current  estimate  for  z,  together  with  a 
penalty  function  for  departure  from  smoothness, 

The  penalty  function  is  mainly  used  to  guide  the  solution  towards  the  minimum. 
In  practice,  the  smoothness  weighting  parameter.  A,  is  slowly  reduced  toward  zero  as 
the  algorithm  converges. 

nun-J  =  £  Jj |  ^£(1)(*  +  ^,y)  -  R{l)(p.q)^j  +  ^£(2)(*  -  £,y)  ~  RU)(p.q)^ 

+  \[z2xx  +  2zlv  + z2yy\}dxdy.  (4.12) 

The  smoothness  term  is  based  on  what  is  called  the  second  variation.  It  is  equivalent 
to  p2z  +  p2  +  q2x  +  g2  when  zx  as  fp/zo  and  zy  as  fq/zo  where  z0  is  the  nominal  depth.2 


JThe  approximations  are  valid  when  the  field  of  view  is  small,  the  image  is  centered  around  the 
camera  s  principal  point,  and  the  depth  of  field  relative  to  the  nominal  depth  is  small 
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Figure  4-6:  Relative  size  of  the  z  array  in  comparison  to  the  image  arrays  for  the  expanded 
boundary  implementation. 

Using  an  array  for  the  discrete  approximation  of  this  cost  function  is. 

"V"-'  -  2 +  S'i,)  - 

+  (f{2)(x  -Yz'y)~Ri2]{p'q]) 

+  X  [-jr  +  2z2zy  +  }  .  (4.13) 

The  F(,)( x  ±  y)  are  interpolated  from  the  input  images  E^(x,  y)  as  in  the  previous 
algorithm. 

4.2.1  Expanded  Boundary  Implementation 

Two  approaches  have  been  taken  to  implement  this  cost  function.  In  the  first  ap¬ 
proach,  the  z  array  is  1  pixel  larger  than  the  image  array  in  all  directions  (see  Fig¬ 
ure  4-6).  The  normal  vector  components  are  estimated  on  the  inner  grid  which  is  the 
same  size  as  the  image  arrays.  In  fact,  the  outer  depth  estimates  are  only  used  when 
computing  p  and  q.  This  approach  is  very  easy  to  implement  but  cannot  be  used  with 
a  multi-grid  scheme  since  the  z  array  is  a  different  size  than  the  image  arrays.  The 
approach  can,  however,  take  full  advantage  of  hierarchical  basis  functions  since  r  can 
be  of  the  size  2”  +  1.  The  implementation  uses  the  vertex- centered  derivatives  dis¬ 
cussed  in  the  previous  algorithm.  The  second-order  derivatives  are  approximated  by 
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the  following  computational  molecules  which  are  also  vertex-centered  approximations. 


"  4c2 


Zx V  “  £2 


2W  -  4f2 


(4.17) 

The  algorithm  based  on  this  approach  has  nearly  the  same  performance  as  the  algo¬ 
rithm  based  on  the  approach  to  be  presented  next. 

4.2.2  Matched  Grid  Implementation 

In  the  second  approach,  z  is  chosen  to  be  the  same  size  as  the  image  arrays  thus 
facilitating  a  multi-grid  implementation.  The  surface  normal  components,  p  and  q. 
are  computed  on  the  boundary  using  the  boundary  extrapolation  technique  discussed 
for  the  zpq  algorithm.  This  approach  also  uses  vertex-centered  approximations  for 
the  derivatives. 

The  performance  of  the  matched-grid  c-only  algorithm  on  the  four  test  images 
is  presented  in  Chapter  5.  This  algorithm  also  gets  stuck  in  the  local  minimum  for 
the  hard  crater  images.  The  performance  on  the  easy  crater  images  and  the  others 
is  very  good  however.  In  fact,  this  algorithm  has  the  best  performance  of  the  all  the 
algorithms  to  be  presented.  An  important  thing  to  note  in  the  performance  figures 
is  that  the  algorithm  converges  to  a  reasonable  surface  quite  quickly,  usuallv  in  the 
first  100  iterations  or  so.  The  performance  figures  are  shown  for  1200  iterations  so 
that  it  is  possible  to  see  the  convergence  characteristics  in  the  long  term. 


1 

-2 

1 

2 

111 

—4 

0 

2 

0 

0 

□ 

m 

0' 

O  3, 

□ 

2  1 

□ 

b 

-4 

0 

m 

0 

□ 

4.3.  DUAL-Z  ALGORITHM 


59 


4.3  Dual-2  Algorithm 


This  cost  function  uses  a  totally  different  variable  representation.  In  this  case  the 
depth  map  is  represented  by  two  arrays.  z(1)  and  z(2)  which  are  estimated  based 
on  the  images  and  E *2)  respectively.  This  algorithm  is  referred  to  as  the  dual-: 
algorithm.  The  two  depth  arrays  are  kept  consistent  with  one  another  using  a  penalty 
function  that  enforces  the  stereo  constraint  equations. 


(4.18) 


where  (x^),  y(1))  are  the  coordinates  of  points  in  the  first  image  and  (t(2),  y(2))  are  the 
coordinates  of  points  in  the  second  image.  These  equations  require  that  the  depth 
estimate  from  one  image  be  the  same  as  the  depth  estimate  at  the  corresponding 
projected  point3  in  the  other  image.  Each  depth  map  is  a  constrained  shape-from- 
shading  solution  for  the  corresponding  image.  This  algorithm  has  a  totally  different 
search  space  than  any  of  the  previous  algorithms. 

For  this  cost  function,  the  depth  maps  are  the  optimization  variables  and 
the  surface  gradients  p(,)  and  q are  computed  directly  from  the  r(,).  The  photo- 
topography  images,  camera  geometry,  and  the  surface  reflectance  functions  are  inputs. 
The  constraints  come  from  the  photo- topography  constraints,  the  integrability  con¬ 
straints.  and  the  stereo  constraints.  The  cost  function  is  formed  by  integrating  the 
squared  shading  error  (in  the  spirit  of  shape-from-shading)  introduced  by  the  current 
estimates  of  c(,),  together  with  penalty  functions  for  departure  from  smoothness,  and 
stereo  error. 


min  J 

2(  1 >.2(2  1 


+  „  (--"V’V')  -  *<»(*'»  -  Jx"<dy<" 

+  \jj  {(£' W11)  -  Rm(p"Kq'»))‘  +  AfV1.-™)2 


where 

V2Z  =  Zxx  +  Zyy.  (4.20) 

The  smoothness  penalty  function  for  this  cost  function  is  based  on  the  squared  Lapla- 


3l'sing  perspective  projection 
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cian  of  z*'*.  The  squared  Laplacian  has  roughly  the  same  characteristics  as  the  second 
variation  [Grimson,  1979],  but  is  much  quicker  to  compute  (since  it  only  involves  one 
convolution). 

The  cost  function  contains  two  shape-from-shading  cost  functions  and  the  stereo 
penalty  function.  This  cost  function  is  not  as  closely-coupled  as  the  other  two  cost 
functions  I  have  described  so  far.  In  particular  the  stereo  constraints  are  added  on 
like  an  addendum  to  the  shape-from-shading  cost  function.  The  stereo  constraints,  in 
effect,  pass  their  constraints  up  to  the  shape  from  shading  parts  of  the  algorithm.  I 
think  of  this  algorithm  as  trying  to  fit  together  two  rubber  mountains.  The  mountains 
can  be  moved  back  and  forth,  and  molded  to  find  the  best  fit.  Since  the  algorithm 
is  not  as  integrated,  we  would  expect  that  images  that  don't  contain  strong  shading 
information  will  be  difficult  for  this  algorithm  to  solve. 

Using  z*1*  and  arrays  as  before,  the  cost  function  can  be  discretized  to  become. 


1 


min  J  = 

2MNe 2 


n<»>«x»>  *■ 


xW.yOJelX 


+ 


£  {  (£<2>(x<2\y<2>)  -  /?(2>(p<2>,9<2>))2  +  A(V2z<2>)2 

n't  ' 


2  MNe2 


c(J),y(J)€TXJ) 


(4.21) 


The  stereo  penalty  functions  include  terms  involving  z(1)  and  z(2)  which  are  inter¬ 
polated  versions  of  z*1*  and  z*2).  Bicubic  interpolation  is  used  [Keys,  1981].  Using 
this  notation,  z*‘*(x,y)  is  the  value  of  z*'1  evaluated  at  the  point  (x,y)  using  bicubic 
interpolation.  Vertex-centered  derivative  approximations  were  used  with  this  cost 
function,  and  the  Laplacian  is  approximated  using  the  computational  molecule. 


Both  the  expanded  boundary  approach  and  the  matched-grid  approach  discussed 
in  the  previous  section  were  taken  to  implement  this  cost  function.  Several  variations 
of  the  basic  implementation  were  tried  with  this  cost  function.  For  example,  in 
addition  to  the  smoothness  term  based  on  the  squared  Laplacian,  algorithms  were 
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implemented  where  the  smoothness  term  was  based  on  the  second  variation. 


min  3 

si  »>.-<*>  2AfAe2 


r(j),vmgDm 


+ 


1 


2  MAV 


E  {••  +  -'  [<*£>’  +  2(42,’)2  +  (42,’)!]  +}  0.23) 


r(2),y(3)gO(S) 


Another  alternate  implementation  used  face-centered  derivatives  instead  of  vertex- 
centered  derivatives.  All  of  these  algorithms  have  performance  very  similar  to  each 
other,  and  they  suggest  that  as  long  as  a  reasonable  derivative  approximation  or 
smoothness  criteria  is  used,  it  doesn’t  really  matter  which  one  is  chosen.  Any  reason¬ 
able  approximation  or  smoothness  criteria  is  as  good  as  any  other.  In  light  of  this,  it 
makes  sense  to  choose  the  approximations  or  smoothness  criteria  that  are  the  easiest 
and  the  fasted  to  compute,  or  that  have  the  best  numerical  properties. 

The  performance  of  this  algorithm  on  four  test  images  is  presented  in  Chapter  5. 
Like  the  previous  algorithm,  this  algorithm  also  has  problems  with  the  hard  crater 
images  but  does  well  on  the  other  images.  Unfortunately,  the  results  of  this  algorithm 
are  two  depth  maps  which  represent  the  depths  as  seen  from  the  right  and  left  cameras 
These  maps  will  not,  in  general,  register  exactly  with  one  another.  To  create  a 
single  depth  map  from  these  maps  requires  some  type  of  averaging.  This  is  a  major 
disadvantage. 

To  address  this  disadvantage,  an  algorithm  was  implemented  that  had  three  depth 
maps  z(1),  r(2),  and  2,  where  z  is  the  depth  map  as  seen  from  the  global  coordinate 
system.  In  all  other  ways  the  algorithm  was  the  same  as  the  dual-2  algorithm.  The 
performance  of  the  algorithm  was  very  similar  to  the  performance  of  the  above  al¬ 
gorithm  yet  it  created  a  central  depth  map.  It  is  not  shown  here  since  it  was  even 
slower  than  the  dual-2  algorithm. 

The  combination  of  two  depth  maps  and  need  for  bicubic  interpolation,  caused 
the  dual-2  algorithm  to  be  the  slowest  to  compute.  The  figures  in  Chapter  5  show 
the  number  of  function  evaluations  and  not  the  computation  time.  In  practice  this 
algorithm  was  3-5  times  slower  than  the  r-only  algorithm. 


4.4  Disparity  Map  Algorithm 


Another  popular  representation  for  depth  that  shows  up  often  in  the  vision  literature 
is  the  disparity  map.  The  disparity  is  the  relative  offset  of  a  point  in  one  image  with 
respect  to  the  corresponding  point  in  the  other  image.  Using  the  notation  we  have 
used  so  far,  the  disparity  can  be  defined  as. 


u  = 


fb 

2z' 


(4.24) 


Li 
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where  u  is  the  disparity.  Using  this  definition  of  disparity,  the  surface  gradient  com¬ 
ponents  are  computed  using, 

»=  /u- 


9  = 


xux  —  u 
/  tty 

yuy-u' 


(4.25) 


The  photo-topography  constraint  equations  can  also  be  re-written  using  disparity  to 
become, 


E^(x  +  u,y)  =  R^(p,q), 
E™(x-u,y)  =  Rl2\p,q). 


Suppose  we  adopt  the  centralized  approach  for  this  cost  function  and  have  the  dis¬ 
parity  map  u  be  the  optimization  variables.  The  surface  gradients  p  and  q  are  com¬ 
puted  directly  from  u.  The  photo- topography  images,  camera  geometry,  and  surface 
reflectance  functions  are  inputs.  The  cost  function  for  this  case  is  formed  by  integrat¬ 
ing  the  squared  photo- topography  error  (based  on  disparity)  together  with  a  penalty 
function  for  departure  from  smoothness, 


mini  =  ^  Jf  {(Eil){x  +  u,y)  -  tfl)(p,q))2  +  (E(2)(x 
+  A[i4  +  2u  ly  +  uly]}dxdy. 


u,y)  -  Rw(p.q)Y 

(4.27) 


where  the  surface  gradient  components  are  given  by  Equation  4.25.  The  smoothness 
term  is  based  on  the  second  variation  of  u.  The  main  difference  between  this  cost 
function  and  the  others  I  have  presented  is  the  smoothness  term  (it  is  now  based  on 
disparity  instead  of  depth).  The  numerical  properties  will  also  be  somewhat  different 
since  the  hierarchical  basis  functions  will  be  over  disparity  instead  of  depth. 

Disparity  is  typically  used  in  vision  algorithms  for  two  reasons.  First  the  disparity- 
based  algorithms  usually  have  fewer  division  operations  which  can  improve  perfor¬ 
mance,  and  second  the  disparity  has  a  much  smaller  dynamic  range  which  can  con¬ 
tribute  to  better  numerical  properties.  Unfortunately,  since  the  photo-topography 
problem  requires  the  use  of  perspective  projection,  the  first  reason  doesn’t  hold  for 
my  algorithms.  The  culprits  are  the  surface  gradient  equations  (Equations  4.25).  As 
for  the  second  reason,  the  disparity-based  DFSS  algorithm  has  pretty  much  the  same 
numerical  properties  as  the  other  DFSS  algorithms.  However,  using  disparity,  instead 
of  depth,  changes  the  solution  space  the  optimization  is  performed  in. 

Representing  u  as  an  array,  the  cost  function  can  be  discretized  to  obtain. 


mjn«J  =  2~Jn?  |(F(1)(i  -I-  u,y)  -  R(1)(p,q))2  +  (F(2,(x  -  u,y)  -  R{2)(p,q))2 


x,y€t> 


+  ^  [UL  +  2u*v  +  uyy] } 


(4.28) 
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The  F{,)(x  ±  u,y)  are  linearly  interpolated  from  the  input  images  E{l)(x.y)  using 
disparity, 

F{i)(x  ±  u, y)  =  £<0(x,y)  +  (x  ±  u  -  x)  [£(,)(x  +  1  ,y)~  E{,)(x.y)]  (4.29) 

where 

x  =  floorfx  ±  u).  (4.30) 

Only  the  matched-grid  approach  was  used  to  implement  this  cost  function.  In 
this  case,  u  is  chosen  to  be  the  same  size  as  the  image  arrays.  The  derivatives 
are  approximated  using  vertex-centered  computational  molecules  as  for  the  previous 
algorithms. 

The  performance  of  this  algorithm  on  four  test  images  is  presented  in  Chapter  5. 
This  algorithm,  like  all  the  previous  algorithms,  gets  stuck  in  a  local  minimum  for 
the  hard  crater  images.  The  algorithm,  however,  performs  well  on  the  other  images. 
An  interesting  observation  can  be  made  about  the  performance  of  this  algorithm  on 
the  mountain  images.  The  algorithm  over-estimates  the  mountain  height  while  the 
r-only  algorithm  under-estimates  it.  This  is  clear  indication  that  the  search  space  for 
the  disparity-based  algorithm  is  fundamentally  different  from  the  search  space  for  the 
r-onlv  algorithm.  Unfortunately,  the  solution  space  is  not  different  enough  to  avoid 
falling  into  the  local  minimum  of  the  hard  crater  image. 


•  • 


•  •  • 


64 


CHAPTER  4.  CANDIDATE  ALGORITHMS 


« 


•  •  • 


Chapter  5 
Test  Results 


It  is  important  when  developing  a  new  algorithm  to  be  able  to  test  its  performance. 

For  computer  vision  algorithms,  that  means  that  it  must  be  possible  to  compare  the 
estimated  surface  with  the  actual  surface.  The  only  way  to  do  this  with  complete 
confidence  in  the  results  is  to  create  synthetic  images  from  a  known  surface  topogra-  • 

phy,  run  the  algorithm  on  these  images,  and  compare  the  estimate  with  the  known 
topography.  Only  after  it  is  shown  that  the  algorithm  works  on  the  synthetic  im¬ 
ages,  can  we  be  confident  that  the  surface  estimates  for  actual  planet  images  will  be 
accurate. 

I  have  created  four  image  pairs  as  test  images  based  on  three  surface  topologies.  • 

•  Easy  Crater  Images.  This  set  of  images  is  based  on  a  crater  on  a  flat  plane. 

The  light  sources  are  oblique  which  results  in  images  that  prove  to  be  easy  for 
the  DFSS  algorithms  to  interpret. 

•  Hill  Images.  This  set  of  images  is  based  on  a  fractally- generated  set  of  rolling  * 

hills.  The  light  sources  are  oblique.  These  images  are  interpreted  correctly  for 

most  of  the  DFSS  algorithms  but  require  more  iterations  of  each  algorithm  since 
they  are  more  complicated  than  the  crater  images. 

•  Mountain  Images.  This  set  of  images  is  based  on  a  fractally-generated  moun-  • 

tainous  terrain.  The  light  sources  are  oblique  and  the  camera  baseline  is  much 

smaller.  This  set  of  images  poses  a  challenge  to  the  DFSS  algorithms  due  to 
the  steep  terrain  and  reduced  baseline. 

«  Hard  Crater  Images.  This  set  of  images  is  also  based  on  a  crater  on  a  flat  • 

plane.  The  light  sources  are  almost  directly  behind  the  camera  resulting  in  a 
set  images  that  prove  to  be  difficult  for  the  DFSS  algorithms. 

The  calibration  parameters  for  the  test  images  are  summarized  in  Table  5-1.  The 
table  lists  values  for  the  baseline  distance  b,  camera  focal  distance  /,  nominal  depth 
c0,  and  light  source  vector  components  p^\  q$\  Ps\  <?s  *■  The  focal  distance  and 
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(2) 

Ps 

JV 

9s 

Easy  crater 

500 

-2750 

-997 

0.0038 

2.62 

0.2 

-0.5 

-0.3 

0.1 

Hills 

500 

-12222 

-1000 

0.0011 

3.41 

1.0 

-1.0 

0.3 

0.1 

Mountain 

100 

-2292 

-996 

0.0153 

1.77 

0.5 

0.5 

-0.5 

0.0 

Hard  crater 

500 

-2750 

-997 

0.0038 

2.62 

0.1 

0.1 

-0.1 

0.1 

Table  5-1:  Camera  geometry. 


nominal  depth  are  negative  to  be  consistent  with  a  right  handed  coordinate  system. 
The  baseline  distance  b  and  depth  z  have  the  same  units  (say  miles),  while  the  camera 
focal  distance,  pixel  spacing  and  the  light  source  components  are  based  on  camera 
units  (say  millimeters).  The  table  also  lists  values  for  relative  surface  height  A :/z0 
and  disparity  Au.  These  two  parameters  indicate  the  difficulty  of  the  problem.  Small 
values  of  either  indicate  a  hard  problem.  Note  the  units  for  A u  are  in  pixels  for  a 
65-by-65  problem,  and  /  is  computed  so  that  the  pixel  spacing  is  1  (millimeter). 

All  of  the  test  images  are  noise-free  so  that  the  best  performance  of  each  algorithm 
can  be  tested.  The  performance  of  the  r-only  algorithm  on  images  with  noise  in 
presented  in  Chapter  7.  The  performance  figures  for  each  test  case  include  mesh  plots 
of  the  estimated  surface  at  several  points  during  optimization  as  well  as  estimated 
images  based  on  the  final  surface.  The  mesh  plots  are  of  a  33-by-33  smoothed  (to 
avoid  aliasing)  and  subsampled  version  of  the  65-by-65  estimated  surface.  The  lower 
resolution  mesh  plots  are  used  to  avoid  printing  problems. 

The  performance  figures  also  show  the  number  of  function  evaluations  (iterations) 
computed  by  each  algorithm  during  convergence.  The  number  of  conjugate  gradient 
updates  taken  is  between  one-half  to  one-third  of  the  number  of  function  evaluations 
since  2-3  function  evaluations  are  needed  to  perform  each  update  step.  All  of  the 
algorithms  use  hierarchical  basis  functions  to  enhance  convergence. 


5.1  Easy  Crater  Images 

The  crater  on  a  flat  plane  is  very  simple  surface  and  thus  serves  as  a  good  test 
surface.1  The  camera  and  light  source  geometry  as  well  as  the  true  surface,  light 
source  contour  plots  and  resulting  images  are  shown  in  Figure  5-1.  As  shown  in  the 
figure,  the  baseline  distance  between  the  cameras  is  about  half  the  distance  to  the 
surface  and  the  light  source  (i.e.,  the  sun)  positions  for  the  two  images  differ  greatly 
from  each  other.  As  a  result,  this  set  of  images  has  strong  shading  information  which 
is  very  easy  for  the  DFSS  algorithms  to  take  advantage  of.  The  effects  of  the  lighting 
can  be  seen  in  the  contour  plots  of  the  reflectance  map.  The  reflectance  contours 
are  separated  enough  in  gradient  space  (p-q  space)  so  that  brightness  values  from 


‘Information  on  how  to  generate  this  surface  can  be  found  in  Appendix  C. 


depth 


5.1.  EASY  CRATER  IMAGES 


Reflectance  Function  Contours 


True  images 


Figure  5-1:  Test  images  of  crater  on  flat  plane  (easy  case).  Shown  are  the  camera  geometry 
as  projected  into  the  xz-  and  yz-planes,  the  true  surface  as  a  mesh  plot,  reflectance  function 
contours  for  the  two  light  source  positions,  and  the  left  and  right  synthetic  noise- free  images. 
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one  image  can  easily  constraint  the  possible  set  of  feasible  gradient  directions.2  The 
images  are  65-by-65  pixels  in  size  and  are  noise  free. 

The  Figures  5-2-5-5  show  the  results  of  applying  the  various  DFSS  algorithms 
to  this  test  case.  All  four  algorithms  can  correctly  interpret  these  images  but  the 
c-only  algorithm  performs  best.  The  figures  show  that  the  estimated  images  in  this 
case  very  closely  resemble  the  true  images  and  the  surface  estimates  are  very  good 
except  in  the  lower  left  corner  of  the  surface  (see  the  mesh  plots).  With  the  lighting 
conditions  chosen  for  these  test  images,  this  small  anomaly  has  little  effect  on  the 
estimated  images.  It  is  interesting  to  note  that  the  anomaly  shows  up  for  all  four 
algorithms. 

The  most  interesting  thing  about  this  test  case  is  the  rate  of  convergence  that  is 
obtained  with  the  z-only,  dual-r,  and  disparity  algorithms.  In  those  cases,  a  pretty 
good  estimate  is  obtained  after  only  50  function  evaluations!  This  represents  only 
about  20  updates  since  more  than  one  function  evaluation  is  necessary  per  update 
when  using  conjugate  gradient  optimization. 


2That  is.  there  are  regions  in  the  gradient  space  where  a  given  brightness  value  from  one  image 
strongly  constrains  the  gradient  direction  there 
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Figure  5-2:  Performance  of  the  zpq  algorithm  on  the  easy  crater  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  A  (solid  line)  and  p  (dashed-line)  cost  function  parameters.  Also 
shown  are  images  based  on  the  estimated  surface  and  mesh  plots  of  the  surface  at  different 
points  during  optimization. 


•  •  •  •  • 


Figure  5-3:  Performance  of  the  z-only  algorithm  on  the  easy  crater  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  X  cost  function  parameter.  Also  shown  are  images  based  on  the 
estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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Figure  5-4:  Performance  of  the  dual-r  algorithm  cn  the  easy  crater  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  A  (solid  line)  and  p  (dashed-line)  cost  function  parameters.  Also 
shown  are  images  based  on  the  estimated  surfaces  and  mesh  plots  of  the  surfaces  at  different 
points  during  optimization. 


Figure  5-5:  Performance  of  the  disparity-based  algorithm  on  the  easy  crater  images.  The 
upper-ieft  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  A  cost  function  parameter.  Also  shown  are  images  based  on  the 
estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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5.2  H’ll  Images 

The  second  set  of  images  is  of  an  undulating  surface  similar  to  eroded  hills  and  was 
generated  using  a  fractal  technique.3  Figure  5-6  shows  the  camera  geometry,  true 
surface,  reflectance  contours,  and  test  images  for  this  surface.  This  set  of  images  is 
more  representative  of  the  type  of  terrain  the  DFSS  algorithms  are  likely  fo  encounter. 
As  shown  in  the  figure,  the  left  camera  is  directly  over  the  surface  and  the  right  camera 
views  the  surface  obliquely.  The  light  sources  are  separated,  as  for  the  easy  crater 
image,  resulting  in  images  with  strong  shading  information.  1  find  the  relatively  low 
65-bv-65  resolution  of  these  images  hard  to  interpret  visually  (unlike  the  crater  images 
which  are  easy  to  interpret).  The  DFSS  algorithms,  however,  perform  well  with  these 
images. 

The  Figures  5-7-5-10  show  the  results  of  applying  the  various  DFSS  algorithms 
to  this  test  case.  All  four  algorithms  work  reasonably  well  with  these  images,  but 
the  r-only  algorithm  performs  best.  Due  to  the  complexity  of  the  surface,  this  set  of 
images  requires  more  iterations  of  each  algorithm  to  obtain  a  satisfactory  estimate  of 
the  surface  than  the  easy  crater  images.  The  figures  show  that  the  c-onlv  algorithm 
performs  best,  it  even  obtains  a  very  good  estimate  after  50  iterations!  The  con¬ 
vergence  is  a  little  slower  for  the  dual-r  and  disparity  algorithms;  they  obtain  good 
estimates  after  about  200  iterations.  The  :pq  algorithm  is  the  slowest  and  requires 
about  600  iterations  to  obtain  a  satisfactory  solution.  The  estimated  images  in  all 
cases  match  very  well  the  true  images  shown  in  Figure  5-6. 


information  on  how  to  generate  this  surface  can  be  found  in  Appendix  C 


CHAPTER  5.  TEST  RESCLTS 


Camera  Geometry 


Reflectance  Function  Contours 


True  Surface 


True  images 


Figure  5-6:  Test  images  of  hill.  Shown  are  the  camera  geometry  as  projected  into  the  iz- 
and  planes,  the  true  surface  as  a  mesh  plot,  reflectance  function  contours  for  the  two 
light  source  positions,  and  the  left  and  right  synthetic  noise-free  images. 
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Figure  5-»':  Performance  of  the  zpq  algorithm  on  the  hill  images.  The  upper-left  graph  shows 
the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface  (dashed-line)  plotted 
against  the  number  of  function  evaluations.  The  upper-right  graph  shows  the  history  of  the 
A  (solid  line)  and  n  (dashed-line)  cost  function  parameters.  Also  shown  are  images  based  on 
the  estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 


Figure  5-8:  Performance  of  the  z-only  algorithm  on  the  hill  images.  The  upper-left  graph 
shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface  (dashed- line) 
plotted  against  the  number  of  function  evaluations.  The  upper-right  graph  shows  the  history 
of  the  A  cost  function  parameter.  Also  shown  are  images  baaed  on  the  estimated  surface 
and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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Figure  5-9:  Performance  of  the  dual-;  algorithm  on  the  hill  images.  The  upper-left  graph 
shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface  (dashed-line) 
plotted  against  the  number  of  function  evaluations.  The  upper-right  graph  shows  the  history 
of  the  A  (solid  line)  and  (i  (dashed-line)  cost  function  parameters.  Also  shown  are  images 
based  on  the  estimated  surfaces  and  mesh  plots  of  the  surfaces  at  different  points  during 
optimization. 
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Figure  5-10:  Performance  of  the  disparity-based  algorithm  on  the  hill  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  A  cost  function  parameter.  Also  shown  are  images  based  on  the 
estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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5.3  Mountain  Images 

The  third  set  of  images  is  of  a  highly  mountainous  surface  and  was  created  to  show 
the  performance  of  the  algorithms  on  steep  terrain.  The  steep  terrain  requires  that 
the  baseline  distance  from  the  cameras  be  shortened  so  that  all  the  surface  points 
are  visible  from  both  cameras  (see  Figure  5-11).  Thus  this  set  of  images  can  be 
used  to  test  the  performance  of  the  algorithms  when  the  stereo  baseline  is  small. 
As  shown  in  the  figure,  the  light  source  positions  for  this  set  of  images  are  widely 
separated  and  generate  deep  shadows  on  this  steep  terrain.  The  reflectance  maps 
are  flat  within  a  shadow  so  no  helpful  gradient  is  available  to  the  algorithms.  In 
addition,  knowledge  that  a  particular  pixel  is  in  shadow  only  constrains  the  set  of 
possible  gradient  directions  to  a  sub-plane  of  gradient  space.  Thus,  within  a  shadow 
region,  much  more  influence  is  given  to  the  brightness  values  from  the  other  image. 
This  combination  of  effects  results  in  slower  convergence. 

The  Figures  5-12-5-15  show  the  results  of  applying  the  various  DFSS  algorithms  to 
this  test  case.  The  r-only,  dual-r,  and  disparity  algorithms  work  reasonably  well  with 
these  images,  but  the  2-onlv  algorithm  performs  best.  The  zqq  algorithm  has  trouble 
with  this  test  case.  Due  to  the  complexity  of  this  test  case,  the  algorithms  require 
many  more  iterations  to  achieve  a  satisfactory  solution.  The  r-only  and  disparity 
algorithms  are  able  to  obtain  good  solutions  after  about  600  iterations,  while  1400 
iterations  are  not  enough  for  the  zpq.  The  dual-c  algorithm  obtains  a  reasonable 
estimate  at  about  1200  iterations. 


Figure  5-11:  Test  images  of  mountain.  Shown  are  the  camera  geometry  as  projected  into 
the  xz-  and  yz- planes,  the  true  surface  as  a  mesh  plot,  reflectance  function  contours  for  the 
two  light  source  positions,  and  the  left  and  right  synthetic  noise-free  images. 
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Figure  5-12:  Performance  of  the  zpq  algorithm  on  the  mountain  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper- right  graph 
shows  the  history  of  the  A  (solid  line)  and  fi  (dashed-line)  cost  function  parameters.  Also 
shown  are  images  based  on  the  estimated  surface  and  mesh  plots  of  the  surface  at  different 
points  during  optimization. 
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Figure  5-13:  Performance  of  the  r-only  algorithm  on  the  mountain  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  X  cost  function  parameter.  Also  shown  are  images  based  on  the 
estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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Figure  5-14:  Performance  of  the  dual-z  algorithm  on  the  mountain  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper- right  graph 
shows  the  history  of  the  \  (solid  line)  and  n  (dashed-line)  cost  function  parameters.  Also 
shown  are  images  based  on  the  estimated  surfaces  and  mesh  plots  of  the  surfaces  at  different 
points  during  optimization. 
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Figure  5-15:  Performance  of  the  disparity-based  algorithm  on  the  mountain  images.  The 

upper-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 

(dashed-line)  plotted  against  the  number  -f  function  evaluations.  The  upper-right  graph 

shows  the  history  of  the  A  cost  function  parameter.  Also  shown  are  images  based  on  the  • 

estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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5.4  Hard  Crater  Images 

Like  the  easy  crater  images  presented  earlier,  this  set  of  images  is  also  based  on  the 
crater  on  a  flat  plane.  However  in  this  case,  the  light  source  positions  are  nearly  the 
same  and  are  almost  directly  behind  the  cameras.  The  resulting  images  are  very  bright 
and  look  very  much  the  same  except  for  slight  differences  in  shading  (see  Figure  5- 
16).  The  reflectance  maps  for  this  case  are  nearly  co-incident  which  contributes  to 
a  surface  orientation  ambiguity.  This  set  of  images  represents  a  worst  case  for  the 
DFSS  algorithms  since  the  shading  information  in  the  images  is  weak  and  the  range 
of  brightness  values  is  small.  However,  they  also  have  strong  stereo  correspondence 
information,  which  unfortunately  is  not  readily  utilized  by  the  DFSS  algorithms.  The 
test  images  are  65-by-65  pixels  in  size  and  are  noise  free. 

The  Figures  5-17-5-20  show  the  results  of  applying  the  various  DFSS  algorithms 
to  this  test  case.  All  four  algorithms  get  stuck  in  a  local  minimum.  The  figures 
show  that  the  estimated  images  resemble  the  true  images  of  Figure  5-16  even  though 
the  estimated  surface  doesn’t  match  the  actual  surface.  The  problem  is  that  the 
algorithms  incorrectly  interpret  the  surface  as  concave  when  it  is  actually  convex  (see 
also  the  shape- from-shading  results  in  Chapter  6).  Even  though  the  stereo  information 
in  these  test  images  can  be  used  to  correctly  determine  the  orientation  of  the  surface, 
the  DFSS  algorithms  rely  very  heavily  on  the  shading  information. 

The  surface  orientation  ambiguity  is  a  result  of  having  both  light  sources  directly 
behind  the  cameras.  All  the  test  images  that  I  have  tried  that  have  this  light  source 
geometry  cause  the  algorithms  to  fail. 

The  results  shown  in  the  figures  are  based  on  algorithms  that  employ  the  hier¬ 
archical  basis  functions.  When  I  run  the  algorithms  on  the  hard  crater  test  images 
without  using  the  hierarchical  basis  functions,  the  algorithms  can  correctly  interpret 
the  images.  The  convergence  is  significantly  slower,  however.  Based  on  several  runs  I 
have  made,  it  appears  that  the  explanation  for  this  behavior  is  that  the  hierarchical 
basis  functions  lead  the  algorithm  down  a  particular  path  that  ends  up  in  the  local 
minima.  When  the  nodal  basis  is  used,  the  algorithm  goes  down  a  different  path  that 
is  able  to  bypass  the  local  minimum  (just  barely). 

This  problem  seems  to  be  sensitive  to  initial  condition.  In  fact,  when  I  start  the 
algorithms  from  an  initial  condition  close  to  the  true  surface,  they  do  converge  to  the 
true  solution  as  expected.4 

This  test  case  has  been  the  impetus  for  developing  a  plethora  of  algorithms,  only 
a  few  of  which  are  presented  in  this  thesis.  I  had  hoped  that  developing  an  algorithm 
with  a  totally  different  search  space  would  result  in  one  that  could  solve  this  problem. 
Unfortunately,  all  the  algorithms  I  have  developed  have  gotten  stuck  in  the  same  local 
minima  shown  in  the  figures.  The  other  test  cases  show  that  the  algorithms  perform 
very  well  when  different  lighting  is  used  where  at  least  one  lighting  condition  is  oblique. 


4Of  course,  they  only  achieve  the  exact  solution  when  the  smoothness  parameters  are  set  to  zero 
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Figure  5-16:  Test  images  of  crater  on  flat  plane  (hard  case).  Shown  are  the  camera  geometry 
as  projected  into  the  xz-  and  yr- planes,  the  true  surface  as  a  mesh  plot,  reflectance  function 
contours  for  the  two  light  source  positions,  and  the  left  and  right  synthetic  noise-free  images. 
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Figure  5-17:  Performance  of  the  zpq  algorithm  on  the  hard  crater  images.  The  up¬ 
per-left  gTaph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimate  '  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper- right  graph 
shows  the  history  of  the  A  (solid  line)  and  n  (dashed-line)  cost  function  parameters.  Also 
shown  are  images  based  on  the  estimated  surface  and  mesh  plots  of  the  surface  at  different 
points  during  optimization . 
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Figure  5-18:  Performance  of  the  z-only  algorithm  on  the  hard  crater  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  A  cost  function  parameter.  Also  shown  are  images  based  on  the 
estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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Figure  5-19:  Performance  of  the  dual-z  algorithm  on  the  hard  crater  images.  The  up¬ 
per-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  gTaph 
shows  the  history  of  the  A  (solid  line)  and  n  (dashed-line)  cost  function  parameters.  Also 
shown  are  images  based  on  the  estimated  surfaces  and  mesh  plots  of  the  surfaces  at  different 
points  during  optimization. 
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Figure  5-20:  Performance  of  the  disparity-baaed  algorithm  on  the  hard  crater  images.  The 
upper-left  graph  shows  the  cost  function  (solid  line)  and  RMS  error  of  the  estimated  surface 
(dashed-line)  plotted  against  the  number  of  function  evaluations.  The  upper-right  graph 
shows  the  history  of  the  A  cost  function  parameter.  Also  shown  are  images  based  on  the 
estimated  surface  and  mesh  plots  of  the  surface  at  different  points  during  optimization. 
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Absolute  Error 

Algorithm  Easy  Crater  Hill  Mountain  Hard  Crater 


dual-z 

1.2991 

0.2274 

0.8774 

2.4241 

zpq 

0.2130 

0.0247 

2.7494 

1.5933 

disparity 

0.1668 

0.0251 

0.8383 

0.9241 

o 

D 

'<f 

0.1728 

0.0247 

0.7799 

1.0716 

Relative  Error 

Algorithm  Easy  Crater  Hill  Mountain  Hard  Crater 


dual-z 

0.3614 

0.0968 

0.8436 

1.0078 

zpq 

0.1678 

0.0205 

2.5715 

1.5379 

disparity 

0.1663 

0.0176 

0.7193 

0.9240 

z-only 

0.1721 

0.0175 

0.7177 

1.0078 

Table  5-2:  Absolute  and  relative  surface  estimation  error  by  algorithm  and  test  case  ( 1200 
iterations). 


5.5  Summary 


A  summary  of  running  the  algorithms  on  the  test  images  is  shown  in  Table  5-2  and 
Figure  5-21.  The  table  shows  the  relative  and  absolute  error  between  the  true  and 
estimated  surface  at  the  last  iteration.  The  absolute  error  is  computed  using  the 
formula 


"^abs  ~ 


1 


(o.i ; 


while  the  relative  error  is  computed  using 


Jrel  = 


1 

NM 


£((~-  -  *)  -  (-*  -  ~?*))2 


(5.2) 


where  z*  is  the  true  surface  height  and  z  is  the  average  of  z.  The  figure  shows 
the  absolute  error  normalized  by  the  surface  depth  change  of  the  true  surface  (i.e.. 
(-mar  —  The  figure  can  be  used  to  interpret  the  error  as  a  fraction  of  the  total 

surface  depth  change.  All  the  algorithms  were  run  with  6  hierarchical  basis  levels 
which  is  one  less  than  the  largest  number  of  levels  that  can  be  used  with  65-by-65 
images.  Using  this  number  of  hierarchical  bases  sped  up  the  convergence  of  each 
algorithm  by  a  factor  of  3-5. 

It  is  clear  that  all  of  the  algorithms  had  problems  with  the  hard  crater  images. 
Each  one  got  caught  in  the  local  minimum.  The  reason  for  this  can  be  seen  in  the 
reflectance  contours  of  Figure  5-16.  For  a  given  brightness  level  (i.e.,  along  one  of 
the  contours),  there  are  two  viable  solutions  with  different  surface  orientations.  The 
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Absolute  RMS  Estimation  Error 


Figure  5-21:  Absolute  surface  estimation  error  scaled  by  the  true  surface  depth  change.  The 
depth  change  is  shown  in  parentheses. 


local  minimum  has  a  orientation  in  the  “dipped"  region  that  is  a  viable  but  incorrect 
orientation,  and  the  stereo  information  is  not  strong  enough  to  pull  it  out  of  the  local 
minimum. 

The  hard  crater  images  have  been  the  driving  force  behind  much  of  this  research 
and  are  one  of  the  reasons  that  so  many  alternate  algorithms  were  implemented. 
As  each  algorithm  was  created  and  implemented  it  was  hoped  that  the  new  search 
space  would  allow  the  correct  interpretation  of  the  hard  crater  images.  Alas,  every 
algorithm  I  have  tried  has  been  caught  in  the  local  minimum. 

The  only  way  the  algorithms  can  correctly  interpret  the  hard  crater  images  is  to 
perform  the  optimization  with  no  hierarchical  basis  levels.  The  convergence  is  very 
slow  but  the  solution  is  correct.  I  hesitate  to  recommend  that  all  runs  be  performed 
using  the  nodal  basis,  however,  since  the  performance  hit  is  huge  and  there  is  no 
guarantee  that  the  true  solution  will  be  found.  Consider  the  fact  that  the  test  images 
are  small  and  that  more  normal  size  images  are  of  the  order  513-by-513.  One  function 
evaluation  with  these  images  will  take  64  times  as  long  to  compute  as  the  test  images. 
In  addition,  since  all  of  the  algorithms  are  diffusion-type  algorithms,  we  could  expect 
an  8  times  increase  in  the  number  of  function  evaluations  to  obtain  convergence  if  the 
nodal  basis  is  used.  As  you  can  see,  the  ramifications  of  not  using  the  hierarchical 
bases  are  huge. 

Fortunately,  the  fact  that  the  algorithms  are  caught  in  a  local  minimum  is  easy 
to  see  for  the  hard  crater  images,  since  the  estimated  images  that  are  created  as 
a  by-product  of  each  function  evaluation  show  streaks  where  the  photo-topography 
constraints  are  not  met  (these  streaks  are  difficult  to  see  in  the  halftoned  reproductions 
of  the  gray  level  images).  Considering  this  fact,  I  recommend  that  the  largest  number 
of  hierarchical  basis  levels  be  used  to  speed  convergence  and  the  results  checked  by 
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9:1 


looking  at  the  estimated  images. 

Clearly,  the  algorithm  that  performed  best  is  the  r-only  algorithm.  Not  only  is 
it  the  quickest  algorithm  to  compute  (i.e..  one  iteration  of  the  r-only  algorithm  com¬ 
putes  faster  that  one  iteration  of  any  of  the  other  algorithms).3  but  it  also  has  the 
best  convergence  rate.  This  algorithm  and  the  next  best  algorithm,  the  disparity 
algorithm,  were  created  by  directly  implementing  the  problem  constraints  as  a  cost 
function.  There  might  be  a  lesson  here  that  the  best  algorithms  are  formed  by  in¬ 
corporating  the  constraint  equations  into  the  cost  function  in  the  most  direct  and 
straightforward  way. 


5The  evaluation  time  for  the  algorithms  is  approximately  proportional  to  the  number  of  free 
variables.  A’2  for  r-only  and  disparity,  3A2  for  zpq.  and  2.V5  for  dual-;. 
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Chapter  6 


Comparison  with 
Shape- from-  Shading 


In  this  chapter  1  show  how  a  simple  shape-from-shading  algorithm  performs  on  my  test 
images.  By  restricting  the  ’-only  algorithm  to  work  on  a  single  image  and  assuming 
orthographic  projection  I  can  create  an  algorithm  similar  to  Szeliski  [Szeliski.  199l] 
and  Leclerc  and  Bobick  [Leclerc  and  Bobick,  1991].  The  resulting  algorithm  is  based 
on  the  cost  function, 

nun  J  =  \jj  {(E(x,y)-R(p,q))2  +  \(V2z)2}dxdy  (6.1) 


where 


q  = 

The  shape-from-shading  algorithm  is  based  on  r  but  cannot  estimate  the  true  depth: 
the  algorithm  can  only  estimate  the  shape.  The  depth  bias  is  unobservable  to  this 
algorithm. 

Following  the  development  of  the  z-onlv  algorithm,  the  discretized  version  of  this 
cost  function  is 

minJ  =  2 ivW  Zv{(E(*-y)-x(M))2  +  W2*)2}-  <6-3> 

Note  that  this  cost  function  directly  uses  the  values  in  E(x,y)  without  having  to 
interpolate  as  was  required  when  forming  F(x,y)  for  the  r-only  algorithm. 

I  show  the  performance  of  the  shape-from-shading  algorithm  on  the  left  image  of 
four  test  image  sets  in  Figures  6-4-6-3.  The  figures  show  the  cost  function  history 
and  optimization  parameters  as  a  function  of  the  number  of  function  evaluations. 
Also  shown  is  the  estimated  surface  shape  at  various  stages  during  the  convergence. 


Figure  6-1:  Performance  of  the  shape- from-shading  algorithm  on  the  left  easy  crater  image. 
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Estimated  rarface  at  ttnnoo  5C 


surface  at  nenDoo  200 


rarface  at  iteration  600 


Estimated  surface  at  deration  1200 


Figure  6-3:  Performance  of  the  shape-from-shading  algorithm  on  the  left  mountain  image. 
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Figure  6-4:  Performance  of  the  shape- from-shading  algorithm  on  the  left  hard  crater  image 
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Image  set 

Rel.  SFS  Error 

Rel.  r-only  Error 

Easy  Crater 

0.6242 

0.1721 

Hill 

0.2083 

0.0175 

Mountain 

1.6343 

0.7177 

Hard  Crater 

1.3532 

1.0715 

Table  6-1:  Estimation  error  of  SFS  algorithm  on  the  test  images  compared  to  r-onlv  results 
( 1200  iterations). 

The  figures  show  that  the  shape- from-shading  algorithm  can  determine  the  shape 
of  the  surface,  but  with  some  error.  I  have  summarized  the  final  error  between  the 
estimated  shape  and  true  shape  in  Table  6-1.  The  estimated  surface  and  true  surface 
offset  so  that  their  average  heights  coincide  before  computing  the  errors  shown  in  the 
table.  The  error  was  computed  using  Equation  5.2. 

The  estimation  error  when  using  shape-from-shading  alone  is  greater  than  the 
estimation  error  when  using  the  z-only  algorithm.  While  this  is  not  surprising  since 
the  r-onlv  algorithm  has  two  images  to  work  with  instead  of  one,  it  is  nice  to  see  that 
significant  performance  gains  are  possible  when  using  a  fused  algorithm. 


Chapter  7 
Error  Analysis 


It  is  equally  important  when  developing  algorithms  to  investigate  their  robustness  in 
the  face  of  errors.  In  this  chapter  I  present  the  results  of  running  the  r-onlv  algorithm 
on  hill  images  that  contain  errors.  I  have  introduced  errors  of  three  types: 

1.  measurement  errors, 

2.  geometry  errors. 

3.  reflectance  errors. 

Each  error  is  introduced  separately  to  simplify  the  analysis.  In  each  case,  a  figure  is 
presented  that  shows  the  relative  and  absolute  error  between  the  true  and  estimated 
surface,  along  with  the  estimated  surface,  convergence  history  and  estimated  images 
for  the  worse  case.  All  the  results  below  are  the  error  after  600  function  evaluations 
(about  300  updates).  The  initial  condition  for  each  case  was  a  flat  plane  at  the 
nominal  depth.  For  reference,  the  relative  and  absolute  errors  after  600  function 
evaluations  on  noise-free  images  are  0.0175  and  0.0247  respectively. 


7.1  Measurement  Errors 

To  determine  the  performance  of  the  2-onlv  algorithm  on  noisy  images  I  added  Gaus¬ 
sian  white  noise  (with  signal-to- noise  ratios  (SNR)  of  100,  10.  or  2)  to  the  hill  images. 
The  resulting  brightness  values  were  then  clipped  to  the  range  (0. 1].  The  results  of 
running  the  r-only  algorithm  on  these  noisy  images  are  shown  in  Figure  7-1  and  Ta 
ble  7-1.  The  figure  shows  that  the  algorithm  performs  well  even  for  a  signal-to- noise 
ratios  of  2  and  that  the  estimate  degrades  gracefully  as  the  SNR  is  decreased. 


7.2  Camera  Geometry  Error 

The  effect  of  errors  in  the  camera  geometry  were  investigated  by  adding  error  to  the 
baseline  distance,  interior  orientation  parameters,  or  by  rotating  the  images  to  move 
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Case  Change  Rel.  Error  Abs.  Error 


Nominal  (c-onlv) 

0.0175 

0.0247 

1 

SNR  100 

0.0203 

0.0275 

2 

SNR  10 

0.0751 

0.1323 

3 

SNR  2 

0.1436 

0.1701 

Table  7-1:  RMS  Estimated  surface  error  from  noisy  images. 


&nm«te4  surface  (case  3)  at  teradoo  601 


Figure  7-1:  Performance  of  z-only  algorithm  on  noisy  hill  images 

the  epipolar  lines  out  of  alignment.  Each  change  was  introduced  independently.  The 
baseline  error  images  were  created  by  changing  both  the  baseline  and  nominal  depth 
by  the  same  percentage  in  order  to  keep  the  interior  orientation  parameters  constant 
( 10%.  20%,  and  30%  changes  where  introduced).  The  interior  orientation  error  images 
were  created  by  offsetting  the  origin  of  the  camera  coordinates  in  a  random  direction 
with  magnitude  of  1,  3,  or  6  pixels.  The  epipolar  error  images  were  created  by  rotating 
the  images  so  the  total  error  introduced  between  the  images  was  0.5,  1,  or  3  degrees. 

The  results  of  running  the  z-only  algorithm  on  these  changed  images  are  shown  in 
Figures  7-2-7-4  and  Table  7-2.  The  baseline  errors  mostly  affect  the  absolute  depth 
of  surface  estimate  and  have  little  effect  on  the  relative  error  of  the  surface.  On  the 
other  hand,  the  small  change  I  introduced  into  the  interior  orientation  parameters 
results  in  a  profound  change  in  the  estimated  surface  (see  Figure  7-3).  Clearly  it 
is  important  to  have  accurate  knowledge  of  the  camera  parameters  to  obtain  good 
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Figure  7-3:  Performance  of  z-only  algorithm  with  hill  interior  orientation  errors 
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Case 

Change 

Rel.  Error 

Abs.  Error 

Nominal  (r-only) 

0.0175 

0.0247 

1 

baseline  error  (  —  10%) 

0.0189 

10.0075 

2 

baseline  error  (—20%) 

0.0374 

99.9331 

3 

baseline  error  (—30%) 

0.0609 

199.8330 

1 

interior  orientation  error  (1  pixel) 

0.0716 

0.0744 

2 

interior  orientation  error  (3  pixels) 

0.3214 

0.3228 

3 

interior  orientation  error  (6  pixels) 

0.9985 

1.0506 

1 

epipolar  error  (0.5°) 

0.0319 

0.0499 

2 

epipolar  error  (1°) 

0.0570 

0.0630 

3 

epipolar  error  (3°) 

0.0943 

0.2432 

Table  7-2:  RMS  Estimated  surface  error  from  geometry  errors. 


estimates. 

Figure  7-4  shows  that  a  moderate  amount  of  error  in  the  epipolar  calibration  can 
be  tolerated  with  only  a  small  effect  on  the  estimated  surface.  This  is  good  news 
since  most  planetary  images  will  have  to  be  re-projected  into  the  aligned  optical  axes 
coordinate  system.  Figure  7-4  shows  that  the  algorithm  can  tolerate  some  errors  in 
this  projection  and  still  produce  meaningful  results. 


7.3  Reflectance  map  errors 

The  effects  of  errors  in  the  reflectance  map  were  investigated  by  changing  the  light 
source  positions  or  by  scaling  brightness  values  in  the  images  to  simulate  an  error 
in  albedo  calibration.  I  did  not  investigate  the  effects  of  using  the  wrong  reflectance 
(such  as  Minnaert  reflectance)  since  the  estimation  errors  due  to  this  effect  cannot  be 
generalized.  The  light  source  error  images  were  created  by  changing  the  light  source 
direction  by  5,  15,  or  30  degrees  in  a  random  direction.  The  albedo  error  images  were 
created  by  scaling  the  images  by  1/0.99.  1/0.95,  or  1/0.90.  The  results  of  running  the 
r-only  algorithm  on  these  changed  images  are  shown  in  Figures  7-S-7-6  and  Table  7-3 
Figure  7-5  shows  that  light  source  position  errors  of  up  to  30  degrees  have  relatively 
little  effect  on  the  estimated  surface.  While,  typical  photo-topography  images  have 
light  source  (sun)  positions  that  are  known  to  high  precision,  these  results  show  that 
the  r-only  algorithm  could  be  applied  to  images  where  the  light  sources  are  not  as 
well  known. 

On  the  other  hand,  Figure  7-6  shows  that  the  algorithm  can  tolerate  albedo  cali¬ 
bration  errors  up  to  10%.  These  results  are  misleading,  however,  since  the  algorithms 
fail  to  converge  when  the  albedo  error  is  greater  than  10%.  The  algorithm  may  not 
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Figure  7-5:  Performance  of  z-only  algorithm  with  hill  light  source  errors. 
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Case  Change _ Rel.  Error  Abs.  Error 


Nominal  (c-only) 

0.0175 

0.0247 

1 

light  source  error  (5°) 

0.0551 

0.1912 

2 

light  source  error  (15°) 

0.0178 

0.3219 

3 

light  source  error  (30°) 

0.0922 

1.1150 

1 

albedo  error  (1%) 

0.0241 

0.0341 

2 

albedo  error  (5%) 

0.0681 

0.0686 

3 

albedo  error  (10%) 

0.1234 

0.1253 

Table  7-3:  RMS  Estimated  surface  error  from  reflectance  errors. 


converge  in  those  cases  since  the  Lambertian  reflectance  map  cannot  exceed  1 .0  and 
the  error  images  produced  contain  many  brightness  values  beyond  that  limit.  The  al¬ 
gorithm  can  probably  tolerate  albedo  calibration  errors  that  don’t  create  normalized 
brightness  values  greater  than  1.0. 


7.4  Summary 

The  foregoing  plots  and  tables  indicate  the  c-onlv  algorithm  is  fairly  robust.  In 

particular,  they  indicate  that  the  algorithm  can  produce  a  reasonably  good  estimate  • 

even  if  the  images  are  noisy,  the  camera  geometry  is  not  known  perfectly,  and  the 

reflectance  properties  are  in  error.  They  also  show  that  it  is  very  important  to  have 

accurate  internal  orientation  parameters,  and  an  accurate  baseline  in  order  to  estimate 

the  depth  correctly. 


•  • 
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Iteration 


Estimated  office  (case  3)  at  tendon  601 


Estmated  mages  (case  3>  at  iteration  601 


Figure  7-6:  Performance  of  2-only  algorithm  with  hill  albedo  errors. 
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Chapter  8 

Algorithm  Extensions 


In  this  chapter  I  discuss  two  ways  of  extending  the  algorithm  to  varying  albedo 
surfaces.  While  all  the  previous  algorithms  have  been  restricted  to  surfaces  with  a 
constant  (known)  albedo,  the  algorithms  discussed  in  this  chapter  work  for  surfaces 
that  have  markings  or  striations.  The  new  algorithms  still  require  that  the  geometric 
reflectance  properties  be  constant  and  known  for  the  whole  surface. 

I  also  show  in  this  chapter  how  more  general  camera  geometries  can  be  accommo¬ 
dated.  Basically,  the  images  are  projected  into  a  coordinate  system  that  has  aligned 
optical  axes. 


8.1  Varying  albedo  algorithms 


As  mentioned  in  Section  2.5.  the  simplification  of  constant  albedo  severely  restricts 
the  applicability  of  the  algorithms  that  are  developed  in  Chapter  4.  In  this  section.  I 
lift  that  restriction.  The  algorithms  that  result,  do  converge  to  a  solution  close  to  the 
actual  surface,  but  the  convergence  is  slower  than  for  the  constant  albedo  algorithms. 

The  first  thing  to  understand  is  whether  we  would  expect  a  varying  albedo  algo¬ 
rithm  to  work.  In  other  words,  'Does  it  seem  reasonable  that  a  unique  value  of  albedo 
can  be  chosen  for  each  point  in  the  image’?  Let's  investigate  that  question. 

Each  point  in  the  images  provides  two  constraints  via  the  two  ■ itended  photo- 
topography  equations 


e[2){i  -  Y^y)  =  p(x^y)^3){p,q). 


(8.1) 


for  the  two  unknowns  r  and  p.  Since  this  is  a  situation  with  two  equations  and 
unknowns,  a  solution  is  at  least  conceivable.  Investigating  further  we  find  that  for 
a  given  p{x,y)  there  are  at  most  two  gradient  directions  (p(x,y),q(x,y))  that  can 
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satisfy  each  equation. 1  These  gradient  directions  cannot  be  chosen  arbitrarily  since 
they  must  be  consistent  with  the  underlying  height  r.  via  the  integrability  constraint 
equations. 


P  = 

<?  = 


•r-r  +  - ' 

f=y 

y=y  +  -  ’ 


(S.2) 


A  solution  is  obtained  when  a  common,  consistent  gradient  estimates  exist.  Con¬ 
versely.  a  given  c  defines  the  gradient  components  and  the  relationship  between  points 
in  the  images.  A  solution  is  obtained  when  p  can  be  chosen  to  match  the  images  E{,) 
with  the  images  estimated  from  the  reflectance  maps. 

From  the  foregoing  discussion  it  should  be  clear  that  it  is  possible  to  create  image 
sets  that  are  inconsistent  so  that  no  solution  exists.  On  the  other  hand  it  should 
also  be  clear  that  given  consistent  images  a  solution  exists.  Thus  we  find  that  the 
extended  problem  is  well-posed. 


8.2  Minimizing  departure  from  a  constant  albedo. 

The  first  algorithm  is  applicable  to  slowly  varying  albedo  surfaces.  It  is  based  on  the 
c-only  cost  function  but  includes  a  penalty  term  of  the  form  p(p  -  p)2  where  p  is 
the  average  albedo  over  the  whole  image.  This  additional  term  penalizes  departure 
from  a  constant  albedo  and  can  be  used  to  estimate  the  calibration  factor  A p  on  the 
reflectance.  The  cost  function  including  this  term  is 


minJ=i//|  .y)  -  p(x.  y)R{l)(p.  q)^j 

+  ^ E{2)(x  -  jpy)  -  p(x,y)R(2){p,q)j 
+  A  [z2„  +  2z2ry  +  4,]  +  p(p  -  p)2}  dx  du  (8.3) 


where 


Iv  pdx  dy 

Iv  dl  dy 


(8.4) 


and  T>  is  the  whole  image.  The  smoothness  penalty  term  is  used  to  guide  convergence 
and  is  not  required  to  guarantee  a  unique  solution.  Usually,  A  is  slowly  reduced  to 
zero  as  the  solution  is  reached  to  avoid  biasing  the  solution.  The  algorithm  based 
on  this  cost  function  converges,  albeit  slowly,  and  works  best  for  surfaces  that  have 
nearly  constant  albedo. 


'For  surfaces  with  Lambertian  Reflectance 


* 
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8.3  Minimizing  local  albedo  change. 

The  second  algorithm  is  applicable  to  surfaces  with  piecewise  constant  or  piecewise 
linear  albedo.  In  this  case,  the  cost  function  includes  a  penalty  term  of  the  form 
p(p  -  p)2  where  p  is  the  average  in  some  local  neighborhood  A*.  This  term  is  similar 
to  a  discrete  approximation  to  the  Laplacian  and  penalizes  departure  from  a  local  av¬ 
erage.  For  a  piecewise  constant  albedo,  this  term  will  be  zero  except  on  the  boundary 
between  the  constant  areas  of  albedo.  The  cost  function  including  this  term  is 

+  ^£<2)(x  -  ~^.y)  -  p(x.y)R(2)(p.q)'j 
+  A  [c*x  +  2z]y  +  r*y]  +  p(p  -  p)2J  dxdy  (8.5) 


where 

_  pdx  dy 

P  c  j  j  ( 8-0 ) 

Ja'  dx  ay 

Following  the  implementation  of  the  z-onlv  algorithm,  the  cost  function  can  be 
discretized  to  become. 


min  J  = 


2N  Mt2 


F(l\x.y)  ~  p(x.y)R{X)(p.q)y 


+  (Fi2){*-y)  -  p(^^y)£{2](p-y)Y 

+  A  [Sl  +  2 :2zy  +  cyy]  +  p(p  -  p)2] 


where  p  is  the  average  in  a  3-by-3  neighborhood. 


(8.T) 


(8.8) 


The  performance  of  this  algorithm  on  the  varying  albedo  test  images  is  shown  in 
Figures  8-2-8-4. 
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8.4  Varying  albedo  test  results. 

To  test  the  algorithm  based  on  this  cost  function,  two  of  the  test  images  were  modified 
to  include  either  a  p  —  0.7  albedo  strip  from  the  lower  left  to  the  upper  right  (Figure  8- 
1)  or  an  p  =  0.7  albedo  variation  in  a  layer  similar  to  what  would  be  expected  from 
sedimentation  processes  (Figure  8-3). 

8.4.1  Crater  with  dark  stripe 

The  first  set  of  test  images  has  the  p  =  0.7  albedo  stripe  imposed  on  the  easy  crater 
test  case.  The  light  source  geometry,  true  surface,  and  true  albedo  are  shown  in 
Figure  8-1.  The  albedo  variation  is  "painted”  on  the  crater  surface. 

The  Figure  8-2  shows  the  result  of  applying  the  varying  albedo  algorithm  to  this 
test  case.  The  figure  shows  the  cost  function  history  and  optimization  parameters  as  a 
function  of  the  number  of  function  evaluations.  Also  shown  are  the  estimated  surface 
and  albedo  image  at  various  stages  during  the  convergence.  While  the  algorithm 
does  converge  to  a  surface  and  albedo  map  close  to  the  true  values,  the  convergence 
is  much  slower  than  for  the  constant  albedo  algorithm  and  the  final  surface  estimate 
has  more  error.  Of  course,  the  constant  albedo  algorithm  would  not  perform  any 
better  if  presented  with  the  varying  albedo  images. 

8.4.2  Hill  with  sedimentation 

The  other  set  of  varying  albedo  test  images  is  based  on  the  hill  test  case  with  the 
albedo  set  to  p  =  0.7  for  surface  heights  in  a  certain  range  (see  Figure  8-3).  The  light 
source  positions  and  geometry  are  the  same  as  for  the  hill  test  case. 

Figure  8-4  shows  the  result  of  applying  the  varying  albedo  algorithm  to  this  test 
case.  The  figure  shows  the  cost  function  history  and  optimization  parameters  as  a 
function  of  the  number  of  function  evaluations.  Also  shown  is  the  estimated  surface 
and  albedo  image  at  various  stages  during  the  convergence.  While  the  algorithm 
does  converge  to  a  surface  and  albedo  map  close  to  the  true  values,  the  convergence 
is  slower  than  for  the  constant  albedo  algorithm  and  the  final  surface  estimate  has 
more  error. 

8.4.3  Summary 

The  varying  albedo  algorithm  can  successfully  estimate  both  the  surface  depth  and 
the  albedo  variation  but  requires  at  least  2  times  as  many  function  evaluations  than 
the  constant  albedo  .-only  algorithm.  In  fact,  the  performance  figures  in  this  section 
show  surfaces  that  have  not  converged  even  after  2800  function  evaluations.  However, 
these  algorithms  must  be  used  for  surfaces  that  cannot  be  estimated  correctly  with 
the  c-onlv  algorithm  because  of  varying  albedo. 


Figure  8-1:  Crater  on  flat  plane  with  varying  albedo.  Shown  is  the  camera  geometry  as 
projected  into  the  xz-  and  y-r-planes,  the  true  surface  as  a  mesh  plot,  albedo  image,  and 
reflectance  function  contours  for  the  two  light  source  positions. 


114 


CHAPTER  8.  ALGORITHM  EXTESSIOXS 


Estimated  images  at  iteraooo  2801 


Ettpittd  surface  at  iteration  201 


Estimated  albedo  at  tferanoo  201 


Figure  8-2:  Performance  of  the  varying  albedo  algorithm  on  the  varying  albedo  crater 


8.4.  VARYING  ALBEDO  TEST  RESULTS. 
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True  images 


True  albedo 


Figure  8-3:  Test  images  of  hill  with  varying  albedo.  Shown  is  the  camera  geometry  as 
projected  into  the  xz-  and  ya-planes,  the  true  surface  as  a  mesh  plot,  albedo  image,  and 
reflectance  function  contours  for  the  two  light  source  positions. 
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Figure  8-5:  Camera  reprojection  geometry. 

The  slow  convergence  is  a  direct  result  of  increasing  the  number  of  degrees  of 
freedom  available  to  the  algorithm.  It  has  the  choice  of  meeting  the  photo-topography 
constraints  by  changing  the  depth  or  by  just  changing  the  albedo.  Initially  it's  easier 
to  change  the  albedo  since  it  shows  up  directly  in  the  constraint  equations.  However, 
using  albedo  changes  alone  is  not  sufficient  to  minimize  the  equations,  so  after  a  while 
the  surface  height  begins  the  change. 

The  convergence  would  probably  be  faster  if  the  stereo  part  of  the  algorithm  was 
more  powerful.  The  disparity  of  the  albedo  edges  should  provide  a  strong  constraint 
on  the  height.  The  fact  that  they  don’t  is  due  to  the  weakness  of  the  stereo  part  of  the 
algorithm.  Hopefully  ongoing  research  will  turn  up  a  more  powerful  stereo  algorithm. 


8.5  General  Camera  Geometry 

The  algorithms  presented  thus  far  are  based  on  the  simplification  that  the  camera 
optical  axes  are  aligned.  This  is  never  true  for  images  taken  from  moving  vehicles 
such  as  inner-planetary  probes,  satellites,  or  aircraft.  The  simplification  of  aligned 
optical  axes  is  used  since  it  greatly  simplified  the  equations.  The  simplification  can 
be  lifted  by  re-projecting  the  images  from  their  true  camera  coordinates  into  the 
coordinates  of  a  virtual  camera  with  its  principal  point  in  the  same  position  but 
with  the  aligned  orientation  (see  Figure  8-5).  I  refer  to  the  true  camera  coordinates 
as  the  primed  coordinate  system  (i.e..  ( x'.y'.:')T  and  the  aligned  coordinates  using 
un-primed  notation,  consistent  with  the  equations  presented  so  far. 

Points  in  a  camera  image  map  to  rays  in  space.  The  mapping  between  coordinate 
systems  preserves  the  orientation  of  these  rays  in  the  global  coordinate  system.  Let 
r'  =  ( x'.y'.f  )T  be  the  coordinates  of  a  point  in  the  camera's  image.  This  maps  to 
the  ray  in  the  direction  r'.  Let  T  be  the  rotation  matrix  relating  the  two  coordinate 


•  •  •  •  •  •  •  •  • 
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systems,  that  is, 


r'  =  Tt, 


(8.9) 


for  any  vector  r  (see  Figure  8-5).  The  ray  in  the  aligned  coordinate  system  is  then 
along  the  direction  Tr'.  This  ray  can  be  re-projected  into  the  virtual  image  plane 
with  focal  length  /  by  normalizing, 


f(TT  r>) 
(TT  r')-z 


(8.10) 


where  z  =  (0.0. 1)7  is  the  unit  vector  along  the 
svstem.  Similarly, 

f(Tr) 

(Tr)-z' 


:  axis  in  the  aligned  coordinate 


(8.11) 


relates  points  in  the  un-primed  image  to  points  in  the  primed  image. 

We  can  use  these  equations  to  create  the  image  that  would  have  been  seen  bv  a 
camera  with  aligned  optical  axes.  Given  E'(r')  in  the  true  camera  coordinate  system, 
the  re-projected  image  is 


£(r) 


(8.12) 


since  TT  is  the  rotation  matrix  from  the  aligned  coordinates  to  the  true  coordinates. 
For  r  =  (x,y,f)T  this  equation  written  in  (x,y)  notation  is 


E(x  y)  =  E'(fnX  +  +  tl3^  t2lX  +  tny  +  ) 

hi*  +  hiy  +  h3f  hi*  +  T32y  +  t^f 


(8.13) 


where  the  elements  of  the  3-by-3  matrix  T  are  t, y  For  discrete  images,  the  value  of 
the  above  expression  can  be  computed  using  some  type  of  interpolation  (say  bilinear 
interpolation). 

Note  that  this  re-projection  can  be  easily  incorporated  into  the  z-onlv,  disparitv- 
based,  and  varying  albedo  algorithms  bv  defining  F(,)  to  interpolate  in  £'  instead  of 
£. 


•  • 
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Chapter  9 
Summary 


This  thesis  has  presented  a  methodology  for  combining  or  fusing  multiple  vision  algo¬ 
rithms.  Four  different  cost  functions  (and  their  associated  algorithms)  were  presented 
that  illustrate  the  methodology.  The  basic  methodology  is  to  combine  the  constraint 
equations  of  the  problem  to  form  a  single  cost  function  in  the  spirit  of  variational 
calculus. 

The  performance  of  the  four  algorithms  was  evaluated  using  four  synthetic  noise- 
free  test  images  of  varying  difficulty.  The  most  closely-coupled  algorithm,  the  --only 
algorithm,  had  the  best  performance.  The  r-only  algorithm  was  able  to  correctly 
estimate  the  synthetic  surface  in  about  200  function  evaluations  for  three  of  the  four 
cases.  For  the  remaining  case  (the  hard  crater  images),  the  algorithm  got  stuck  in  a 
local  minimum,  as  did  all  the  algorithms  tried.  This  case  has  lighting  geometry  that 
results  in  ambiguous  shading  information. 

It  was  shown  that  the  c-only  algorithm  has  much  better  performance  using  the 
two  photo-topographv  images  than  a  simple  shape-from-shading  algorithm  which  uses 
only  one  image.  This  performance  increase  validates  the  fusion  approach  to  obtaining 
better  performing  vision  algorithms. 

The  r-only  algorithm  was  also  shown  to  be  robust;  able  to  accurately  estimate  the 
synthetic  surface  in  the  presence  of  several  types  of  errors.  The  performance  of  the 
algorithm  based  on  images  that  contained  noise,  geometry  error,  or  reflectance  errors 
was  shown.  In  most  cases,  the  algorithm  was  able  to  form  a  good  estimate. 


9.1  Mars  Images 

The  robustness  and  performance  of  the  algorithm  on  synthetic  images,  builds  confi¬ 
dence  that  the  algorithm  can  perform  similarly  on  real  images.  One  such  set  of  real 
images  is  shown  in  Figure  9-1. 1  The  images  are  Viking  stereo  images  of  Mars.  The 
images  as  received  (and  shown)  are  processed  versions  of  the  original  Viking  images. 


‘We  are  grateful  to  Mike  Caplinger  for  providing  these  images 


120 


CHAPTER  9.  SUMMARY 


They  have  been  reprojected  into  an  aligned  coordinate  system  and  filtered  to  remove 
biases  due  to  any  large  scale  albedo  variations.  The  images  also  have  very  similar 
lighting  conditions  (they  were  probably  taken  close  together  in  time).  This  image 
pair  thus  doesn't  represent  the  best  possible  situation  for  the  algorithms  developed 
in  this  thesis.  Nevertheless,  the  r-only  algorithm  performs  well. 

The  estimated  surface  from  the  c-only  algorithm  is  shown  in  Figure  9-2  and  9- 
3.  The  figures  show  that  the  ;-only  algorithm  produces  a  reasonable  estimate 
of  the  Martian  surface.  It  should  be  noted  that  the  camera  geometry  and  light 
source  positions  were  given,  but  the  reflectance  map  for  the  surface  of  Mars  was  not. 
The  results  shown  are  based  on  the  assumption  of  a  Lambertian  reflectance  map. 
The  nominal  slope  of  the  estimated  surface  (see  surface  plots)  is  an  artifact  of  the 
camera  baseline  orientation  with  respect  to  the  planet's  surface  and  can  be  removed 
by  shifting  the  camera  positions  along  their  line  of  sight.  In  fact,  the  estimate  shown 
is  based  on  cameras  that  are  shifted  slightly  so  that  the  light  source  positions  could 
be  represented  using  (p„qa)  gradient  components.2 

While  it  is  difficult  to  see  in  the  halftone  reproduction  of  the  estimated  images, 
the  images  contain  a  slight  ghosting  effect.3  The  ghosting  is  like  a  double  image  and 
is  depth  dependent.  By  running  the  algorithm  with  slightly  shifted  camera  principal 
points,  different  parts  of  the  estinr  ted  images  can  be  brought  into  registration.  The 
ghosting  is  probably  caused  h;  using  Lambertian  reflectance  instead  of  the  true  radi¬ 
ance  function  for  the  Martian  surface.  However,  even  though  the  algorithm  used  an 
inaccurate  reflectance  function,  the  surface  estimate  does  contain  many  of  the  small 
and  large  scale  features  found  in  the  v'iking  images.  Examples  include  the  cliff  in  the 
upper  right  and  the  valley  just  below,  as  well  as  the  craters  near  the  bottom  of  the 
.mages  and  the  ridges  in  the  center.  An  even  better  estimate  of  the  Martian  surface 
could  have  been  obtained  if  the  true  reflectance  function  were  used  with  the  r-onlv 
algorithm. 


9.2  Future  research 

There  are  several  directions  this  research  can  go  in  the  future: 

•  Apply  the  fusion  methodology  outlined  in  this  thesis  to  other  vision  problems. 

•  Apply  the  r-only  algorithm  to  more  real  images  to  further  judge  its  performance. 
Particularly  apply  the  algorithm  to  image  sets  with  differing  lighting  conditions. 

•  Determine  the  performance  of  the  ;-only  algorithm  when  more  realistic  re¬ 
flectance  maps  «,  ^  us  .  with  *he  algorithm.  Recall  that  while  I  only  tested  the 


2  A  requirement  for  the  particular  implementation  I  he-  ■  used  With  slight  mod  ifm  at  ions  the 
algorithms  could  be  used  with  any  other  light  source  position  parameterization 

3The  extent  of  the  ghosting  is  less  then  2  pixels  for  a  257-bj-257  image 
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Camera  Geometry 


True  images 


Figure  9-1:  Stereo  images  of  Mars  taken  by  the  Viking  probe. 
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Figure  9-3:  High  resolution  mesh  and  surface  plot  ol  estimated  Mars  surface.  The  surface 
plot  was  created  by  coloring  the  surface  using  the  left  estimated  image. 
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algorithms  on  surfaces  with  Lambertian  reflectance,  other  reflectance  functions 
can  easily  be  used  with  the  algorithm. 

•  Develop  an  algorithm  with  better  stereo  integration  that  can  correctly  estimate 
the  surface  for  the  Hard  Crater  test  case.  To  do  this  may  require  developing  a 
new  innovative  stereo  algorithm. 

Of  these  possible  directions,  the  last  one  presents  the  most  challenges  and  offers  the 
most  rewards. 


9.3  Conclusions 

This  thesis  has  shown  that  the  variational  approach  to  fusion  problems  can  produce 
robust,  well  performing  vision  algorithms.  It  has  also  shown  that  fused  algorithms 
can  have  significant  performance  gains  over  non-fused  alternatives. 

The  four  example  algorithms,  each  based  on  a  different  variable  representation, 
showed  that  choosing  the  right  variable  representation  is  important  to  achieving  good 
estimates. 

Finally  in  conjunction  with  this  research,  a  new  algorithm,  the  r-only  algorithm, 
was  developed  to  solve  the  photo-topography  problem.  This  is  the  first  well  perform¬ 
ing  algorithm  to  solve  that  problem. 


Appendix  A 

Gradient  derivation  for  --only 
algorithm 
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This  appendix  presents  the  gradient  derivation  for  the  r-only  algorithm  for  use  with 
the  conjugate  gradient  optimization  technique. 


A.l  Cost  function. 


In  the  main  text,  the  discrete  approximation  to  the  cost  function  for  the  r-only 
algorithm  is  given  as 


min  J  =  Y.  |  (/r(,)( y )  -  R{']{p.q))2  +  (Fl2){i.y)  -  R^ip.q))' 


■r.yCP 


+  A  [~«  +  2zly  +  *yy)  } 


(A.l) 


where  V  is  the  discrete  domain  of  the  underlying  variables  in  the  global  coordinate 
system,  p  and  q  are  computed  using 


P  = 

<7  = 


f-r 


•V'y  + 


(A.'Ji 


.1/  and  .V  are  the  row  and  column  dimensions  of  the  discrete  domain  and  t  is  the 
grid  spacing  (assumed  to  the  same  in  both  the  x  and  y  directions).  The  F{,i(i.y)  are 
interpolated  from  the  input  images  £(,)(x,  y)  via  linear  interpolation. 

F(,)(x.y)  =  £H(.r.y)  +  ("J  ±  -  x)  \e{,)(x  +  l.y)  -  E{,)(i.y)\  (A  d) 


•  • 
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where 


x  =  floor(  x 


I  A. 4  i 


and  the  floor(x)  function  returns  the  greatest  integer  that  is  smaller  than  x.  These 
equations  assume  that  x  is  sampled  on  unity-spaced  grid. 


A. 2  Bicubic  extrapolation 

Recall  that  the  matched-grid  implementation  is  used  for  the  x-only  algorithm.  In 
this  implementation,  x  is  the  same  size  as  the  image  arrays  E and  the  boundary  is 
extended  if  necessary  based  on  bicubic  interpolation.  Let  xtl  be  a  value  in  the  first 
column  of  x  (i.e.,  on  the  left  boundary  of  x).  Then  the  extrapolated  value  xl0.  is 
computed  using 

-i0  =  3x,i  —  3xl2  -t-  xl3.  (A. 51 

Similar  equations  work  for  the  other  boundaries  of  z.  In  this  way.  a  M-bv-.Y  matrix 
can  be  extrapolated  to  form  a  (.V/  +  2)-bv-(.Y  4-  2)  matrix. 


A. 3  Gradient  of  terms  based  on  convolution. 

Theorem  1  Let  y  be  computed  via  a  2-D  convolution  from  the  field  z  and  the  filter 
h.  y  =  z  *  h.  Let  f(x)  :  9?  — *  ??.  be  any  scalar  point  function.  Then  the  derivative  of 
the  scalar  cost  function  J  =  Y. iYj  fiy<j)  U’t/A  respect  to  zim  is 


8J  _  df(i) 
dz,m  dr 

where 

htJ  =  h-,.-j. 


(A. 6) 

(A. 7) 


Proof  Using  the  definition  of  the  2-D  convolution. 

y>J  ~  y  .  -ks^t-k.j-s- 

k  J 


(A. 8) 


the  cost  function  can  be  expanded  to  obtain. 


i  ;  V  k  s 
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The  sums  are  taken  over  the  entire  range  of  indices  into  the  matrices.  Taking  the 
derivative  and  using  the  chain  rule  we  find. 

dJ_^2^df  d 


dz 


lm 


dx  dzim 


df 


df 


*  ; 


y .  y*!  -ksh^k.j-s 

.  k  i 

(A. 10) 

-drlm  "*J-5 

j  A .  1 1 1 

• 

-rrj 

I  A.  12) 

-7 

(A.l  31 

• 

(A. 14) 

dx 

t  J 

df  A 
=  Tx*h 

The  result  holds  for  computational  molecules  as  well  since  operating  with  a  com¬ 
putational  molecule  m  is  equivalent  to  convolving  with  m. 

Theorem  2  Theorem  1  also  holds  for  scalar  functions  of  multiple  argument.-.  In  that 

case. 

8J  df  r  .  df 


J  Li  ’ 

~  =  h  +  z~*9- 
d:  dx  i  di2 


(A.  15) 


Proof  Suppose  y  and  u’  are  defined  via  2-D  convolution,  y  ~  z  *  h.  u-  =  c  *  g. 
and  fix !.  j2)  '•  [S  x  £]  —  %t,  then  the  partial  derivative  of  the  cost  function  J  = 
/(y.j  •“'<;)  >s 

dJ  df  r  df 

The  proof  follows  the  same  lines  as  the  proof  for  Theorem  1  I. 


A. 4  Cost  function  derivative. 

I’sing  the  chain  rule,  the  derivative  of  the  cost  function  (Equation  A.l)  is 

dg 


-  «'">  -  *1" 
iZkm  I  „  1,  \  Ct:*m  dzkm 


\  <7*Am 

d:zz 


RivJP-  _ 


d^ 


km 


+  A 


+  2;  dZzy 


+ 


wvv 


-km 


^dZkm  ~>V  dz 


km 


} 


(A.r 
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Let’s  look  at  each  term  in  turn.  The  derivatives  involving  the  interpolated  images 
can  be  computed  using  the  definition  of  F(,). 


dF 

dzkm 


+  ^  ±  ~  (£*(*  +  l.y)~  EAi.y)} 


=  T^Ez((x).y). 


(A. IS) 
(A. 19) 


That  last  equality  is  possible  since  dx/dzkn  =  0  for  one-sided  derivatives. 

The  derivatives  of  the  gradient  components  p  and  q  can  also  be  computed  using 
their  definition.  They  are. 


and. 


dp  fjft Sx-r  +  -)  -  S~r 


dzkm 


l-xf*-  + 

J-krrx 


J  ~r 


(xzz  +  z)2 

3z 


(XZZ  +  2)2 


d<7  _ 

d:km  (yzy  +  z)2 


(A. 20) 
(A.21) 

(A.22) 


As  discussed  in  the  main  text,  the  partial  derivatives  of  z  (cT,  cv.  ;rr.  and  so  on), 
are  computed  via  2-D  computational  molecules.  C ill  these  molecules.  hz.  hy.  hzz. 
and  so  on.  in  the  obvious  way.  Using  the  results  of  Theorem  1.  the  derivative  of  the 
cost  function  can  then  be  written. 


■£L  _  [_(£<»  -  #*>)£<*>  +  (F(2)  -  F(2))£<2)]  It 

Q~km  £Z 

+  [(£ “>  -  +  (F(2)  -  F<2>)£<2>]  /'^ 

+  [(£(l1  -  +  (F(2>  -  /?,2))/^2)] 

-  |  [(F<"  -  R<")R<"  +  IF*2'  -  R<!>)R<p3>  j  {x_f:+  —jo  hz 

-  |  [(F<1>  -  +  (F(2)  -  tf(2>)£<2>]  j-Il  — |  o  hy 

+  A  [s„  o  hzz  +  2 zxy  O  hZy  +  Zyy  o  .  (A. 23) 

These  equations  are  not  the  whole  story  since  they  don’t  work  on  the  boundary.  I 


•  •  •  • 


•  •  • 
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will  show  how  to  take  into  account  the  effect  of  the  bicubic  extrapolation  on  the  left 
boundary:  the  other  boundaries  are  dealt  with  in  a  similar  way.  Suppose  h  has  either 
2  or  3  columns,  and  let  z  be  the  extrapolated  version  of  c  (for  simplicity  assume  c 
is  square  of  size  A -by- A).  In  this  case  the  subscripts  on  c  run  from  0  to  A  —  1. 
Theorem  1  states  that  the  derivative  of  J  =  52  H  /(i  *  h)  is 


( A. 24 1 


Call  this  matrix  G.  Since  i  is  computed  from  r  via  the  bicubic  extrapolation  described 
in  Section  A. 2.  we  can  compute  dJ/dz  by  taking  into  account  the  dependence  of  zM 
on  ct).  zl2.  and  zl3.  Let  G  be  the  matrix  dJ/dz.  then  columns  1-3  of  G  can  be 


computed  using 


G ,i  =  G,i  -t  3(7,0-  { A. 25 1 

Gx  2  =  C  ,2  —  3G,o.  (A. 26) 

(7,3  =  6i3  +  G,o-  (A. 27) 


Similar  expressions  hold  for  the  other  boundaries. 


i 


( 


( 


< 


4 


4 
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Appendix  B 
M-file  Listings 


This  appendix  contains  the  MaTLAB1  M-file  source  files  for  the  DFSS  algorithms. 
The  source  files  are  presented  for  the  six  algorithms  in  this  thesis:  zpq.  r-only.  dual 
r.  disparity,  sfs.  and  varying  albedo.  A  key  to  the  files  is  given  in  Table  B-l. 


B.l  Cost  function  and  gradient  routines 


B.l.l  hbdfss_cost3c.m 

1  function  [ J  ,FR1  ,FR2]  *hbdfss_cost  (v  .zsiza  .lavals  .paraas  ,E1  ,E2  ,  laabda.au ) 

2  XHBDFSS.C0ST3C  Coat  function  for  dapth  from  shading  and  atarao 

3  X  problOB  using  hiararchical  basis  functions.  Usos  trua 

4  X  parspactiva  projaction. 

6  X  J“MBDFSS_C0ST3C(V .ZSIZE, LEVELS , PARAKS ,El ,E2 .LAHBDA ,HU>  uhara 
-  X  V  •  [z.p.q]  ara  tha  optimization  variablas,  ZSIZE  is 
s  X  SIZE(z),  LEVELS  is  tha  nuabar  of  h-basis  lavals.  El  and  E2  ara 

9  X  tha  input  inagas.  LAHBOA  is  tha  scalar  saighting  factor  on 

10  X  dapartura  fron  saoothnass  and  HU  is  tha  scalar  saighting  factor  on 

11  X  intagrability  Tha  iaaga  paraaatars 

12  X  PARAHS  a  £f .b.zO.psl ,qsl ,ps2.ps2, vzl ,vyl ,vz2,vy2j . 

13 

14  X  Clay  H.  Thompson  S-18-92 

16  X  lavissd  to  usa  corract  p.q  calculation 
16  X  to vi s ad  to  usa  p.q.z  of  tha  sasw  siza  as  E. 
l-  X  Ravisad  to  output  astiaatsd  iaaga 
is  X  Ravisad  to  support  aulti-grid  achaaa 

19 

20  X  Caaara  constants 

21  f  •  paraas(l) ; 

22  b  •  paraas  ( 2 ) ; 

23  zO  •  paraas (3); 

24  gaaaa  •  f»b/2; 

26  dalta  ■  paraas ( 12 ) , 

26  araa  •  prod(zsiza)*daltaadalta; 

23 

28  X  Light  Sourca  positions 

29  pal  •  paraas(4);  qsl  •  paraas(S) : 

30  ps2  •  paraas (6) ,  qs2  •  paraas (7 ) ; 

31 

32  X  Caaara  coordinata  calibration 

33  vzl  •  paraas (8) ;  vyl  •  paraaw(9), 

34  vz2  •  paraas(lO);  vy2  •  paraas(ll), 


'MaTLAB  is  a  product  of  The  Math  Works,  Inc  .  24  Prime  Park  Way.  Cochituate  Place  Natick 

MA  01760.  (508)653-1415. 
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hbdfss-Cost3c.m 

hbdfss-grad3c.m 

hbdfss-Cost2c.m 

hbdfss.grad2c.m 

hbdfss_cost4.m 

hbdfss-grad4.m 

hbdfss_cost7.m 

hbdfss-grad7.m 

hbdfss_cost8b.m 

hbdfss_grad8b.m 

hbsfs.cost.m 

hbsfs grad.m 


Algorithm  cost  functions  and  gradient  routines _ 

Cost  function  for  zpq  algorithm. 

Gradient  function  for  zpq  algorithm. 

Cost  function  for  2-only  algorithm. 

Gradient  function  for  z-onlv  algorithm. 

Cost  function  for  dual- 2  algorithm. 

Gradient  function  for  dual-2  algorithm. 

Cost  function  for  disparity  algorithm. 

Gradient  function  for  disparity  algorithm. 

Cost  function  for  varying  albedo  algorithm. 

Gradient  function  for  varying  albedo  algorithm. 

Cost  function  for  shape-from-shading  algorithm. 
Gradient  function  for  shape-from-shading  algorithm. 


_  Support  routines _ 

rmap  Lambertian  reflectance  function.  ~ 

rmapp  Derivative  of  reflectance  function  with  respect  to  p. 

rmapq  Derivative  of  reflectance  function  with  respect  to  q. 

conjgrad.m  Conjugate  gradient  optimization. 

lsearch.m  Line  search  function  for  conjugate  gradient  optimization. 

filter2d.m  2-D  computational  molecule  filtering. 

cfilter2d.m  2-D  computational  molecule  filtering  with  bicubic  interpolation. 

hbasis.m  Main  level  hierarchical  basis  conversion. 

hb.m  Hierarchical  basis  interpolation. 

hbt.m  Adjoint  hierarchical  basis  interpolation. 

interpx.m  Linear  interpolation  in  the  x  direction. 

domain2d.m  2-D  plaid  domain  generation. 

icubic.m  1-D  cubic  interpolation. 

dcubicx.m  Derivative  of  1-D  cubic  interpolation  with  respect  to  x. 

dcubicz.m  Derivative  of  1-D  cubic  interpolation  with  respect  to  z. 


Table  B-l:  M-file  Descriptions. 
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m 


35  vz  *  (vzl*vz2)/2.  vy  ■  (vyl*vy2>/2 : 

36 

3-  X  Extract  z,p,q  and  tranafora  into  nodal  baziz 

38  nz  »  uuo(l);  n z  »  zalza(2), 

39  z  «  v( :  ,1 :  nz )  ; 

40  p  «  v(  : ,nz*[l  nz] > ; 

41  q  *  »<  :  ,2»nz  *  [lnz]>. 

42  z(  >  *  hbaaia(z .lavalz) , 

43  p(  )  •  hbaai»(p .lavalz) ; 

44  q(  )  •  hbaaiz(q, lavalz) ; 

45 

46  X  Spacial  coordinataa  in  iaaga 
4'  [nz .nz]  •  ziza(z) ; 

4g  [x,f]  *  doaain2d( [0.az-lj«daltaa*z*l , [0: nz-l]«dalta*vy*l ) ; 

49 

50  X  Stancilz 

51  hzz  »  [-1  0  l]/(2»dalta) ,  X  X-darivativa  for  zz 

52  hyz  *  [1 ,0 ; -1J/ (2*dalta) ;  I  Y-danvativa  for  zy 

53  hz  »  [-1  1 : -1  l]/(2*dalta) .  X  I-darivativa 

54  hy  «  II  1 ; -1  -lj/(2adalta) ,  X  Y-darivativa 

55 

56  X  Coaputa  raflactanca  nap  valuaa . 

5"  El  *  map(p.q.pal  ,qal)  , 

56  R2  =  raap<p,q,pz2 ,q*2) ; 

59 

60  X  Coaputa  nuaancal  darivativaa  of  p,q,  and  z. 

61  pz  •  f iltar2d(p,hx,  ’raziza ’ ) ; 

62  py  *  filtar2d(p,hy ,  ’raziza ’) ; 

63  qz  >  filtar2d(q,hx, ’raziza ’) ; 

64  qy  ■  filtar2d(q,hy,  raziza’); 

65  zz  *  cf iltar2d(z,hzz) ; 

66  zy  »  cfiltar2d<z,hyz) ; 

6' 

66  X  Coaputa  aatinataz  of  p,q  uzing  nuaarical  darivativaa  of  z 
69  pa  »  f»ZZ  /(z.azzaz) ; 
ro  qa  »  f»zy ./(y  azy+z) ; 

71 

72  X  Coaputa  dizparity 

73  d  »  (fab/2)  /z, 

74 

75  X  Dataraina  ztarao  aappad  laagaz  El ,F2  Sat  arror  to  zaro  vhara  F  iz  lal 

76  FI  »  intarpx(El , (xad-vxl-1 )/dalta*l ) ; 

out  «  iannn(Fl),  if  any(out(  >),  Fl(out)  «  El(out),  and, 

76  F2  »  intarpx(E2 , (z-d-vx2-l )/dalta+l ) , 

79  out  a  ianan(F2).  if  any(out(  )),  F2(out)  •  E2(out),  and, 

60 

81  teral  «  (0  5/araa) azua(zua( (Fl-Rl )  "2  *  <F2-R2>  *2)) 

82  tara2  »  (0  S/araa)alaabdaazua(zua(pz  *2  ♦  py  ‘2  ♦  qz  *2  ♦  qy  ‘2>), 

63  tam3  •  (0  5/araa  >  aau«zua(  zua(  (pa-p ).  ‘2  ♦  (qa-q).'2>). 

84  Xdizp(apnntf  ( ’Taraa  :  X12  Sf  X12  Sf  X12  5f  ’  ,  taral ,  tara2  ,  taraS) )  , 

85  J  •  Ctaraiatara2+tara3,taral  ,tara2,tara3]  , 

86 

6'  if  nargoutil. 

66  FR1  a  [El ; FI ; R1 ; aba (Fl-Rl )] ; 

89  F12  »  [E2 ;F2 ,R2 ;aba(F2-R2) J ; 

90  and 

91 

B.1.2  hbdfss_grad3c.m 

1  function  [J.FE1 ,FR2]»hbdfaz.coat (v .zaiza .lavalz .paraaz ,E1 , E2 .laabdz ,au) 

2  XMBDFSS.C0ST3C  Coat  function  for  dapth  froa  ahading  and  ztarao 

3  X  problaa  uzing  hiararchical  basiz  function*  Uzaz  trua 

4  X  parzpactiva  projaction 

5  X 

6  X  J»HBDFSS_C0ST3C(Y,ZSIZE. LEVELS, PAAAMS , El. E2 .LAMBDA .MU)  zhara 
*  X  V  a  [z,p,q]  ara  tha  optiaization  variablaa,  ZSIZE  iz 

6  X  SIZE(z),  LEVELS  iz  tha  nuabar  of  h-baziz  lavalz  El  and  E2  ara 

9  X  tha  input  laagaz,  LAMBDA  iz  tha  acalar  aaighting  factor  on 

10  X  dapartura  froa  zaoothnaza  and  MU  iz  tha  acalar  uaighting  factor  on 

11  X  intagrability .  Tha  iaaga  paraaatarz 

12  X  PARAMS  ■  [f ,b,zO,pzl ,qzl ,pz2,pz2,vzl ,vyl ,vx2 , vy2] 

14  X  Clay  M  Thonpaon  5-18-92 

15  X  Ravizad  to  uaa  corroct  p,q  calculation 

16  X  Ravizad  to  uaa  p,q,z  of  tha  zaaa  aizz  aa  E 

17  X  Ravizad  to  output  aztiaatad  iaaga 

16  X  Ravizad  to  aupport  aulti-grid  achana 
19 


•  • 
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20  X  Cuan  constant* 

21  f  ■  ptrau(l)  . 

22  b  ■  paraa*(2> , 

23  zO  *  paraas(3) , 

24  giau  *  f*b/2; 

25  dalta  *  para*s(12)  ; 

26  araa  *  prod(zsiza)*dalta*dalta, 

27 

2b  X  Light  Sourca  positions 

29  psl  *  parasa(4);  qsl  *  paraas(S). 

30  ps2  ■  paraas«S);  qs2  *  pnraas(7)  ; 

31 

32  X  CtMrt  coordinata  calibration 

33  vxl  ■  par  asm  (6) ;  vyl  *  paraas(9); 

34  vx2  *  paraas(lO).  ry2  *  paraas(ll); 

35  vx  «  (vxl+vx2)/2;  try  *  <vyl*vy2)/2, 

36 

37  X  Extract  z,p,q  and  traasfora  into  nodal  basis. 

38  az  *  zsiza(l) ;  nz  *  zsizo(2); 

39  Z  *  tr(  :  ,1  nz)  ; 

40  p  *  v( . ,nz*£l  nz] > ; 

41  q  *  v(:,2*nz  ♦  [l.fiz]); 

42  z ( : )  *  hbasis (z , lavals )  ; 

43  p( : )  *  hbasis (p , lavals) ; 

44  q( : )  *  hbasis (q , lavals) ; 

45 

46  X  Spacial  coordinatss  in  isiags . 

4?  fax.nz]  *  siza(z) ; 

48  [x,y]  *  doaain2d( [Oaz-ljadalta+vx+i , [unz-ij^dalta+vy+l) ; 

49 

50  X  Stoncils 

51  hxz  *  F-l  0  ll/(2*dalta) ;  X  X-darivativa  for  zx 

52  hyz  *  Cl ;0;-lj/(2*dalta) ;  X  Y-danvativa  for  zy 

53  hx  *  [-1  1;-1  ll/(2*dalta) ;  X  X-darivativa 

54  hy  *  Cl  1;-1  -lJ/(2*dolta) ;  X  Y-darivativa 

55 

56  X  Coaputa  raf lsctanca  aap  valuas 

5?  II  *  raap(p.q,psl ,qsl) ; 

58  12  *  raap(p ,q,ps2 ,qs2) ; 

59 

60  X  Coaputs  nuaarical  darivativas  of  p,q,  and  z 

61  px  *  f lit sr2d(p  ,hx , ’ rasiza ’ ) ; 

62  py  *  filtar2d(p,hy , ’rasiza ’) ; 

63  qx  *  filtar2d(q,hx. ’rasiza*) ; 

64  qy  *  f iltar2d(q,hy , 'rasiza ’) ; 

65  zx  »  cf iltar2d(z,hxz) ; 

66  zy  *  ctilfr2d(z ,hyx>  ; 

67 

68  X  Coaputa  astiaatas  of  p.q  using  nuaarical  darivativas  of  z 

69  pa  *  f*zx./(x»xx*x) ; 

70  qa  »  f *zy/(y*xy*z> ; 

71 

72  X  Coaputa  disparity. 

73  d  »  (fab/2) . /z; 

74 

75  X  Dataraina  starao  aappad  laagas  Fl  ,F2  Sat  arror  to  zaro  vhara  F  is  lal 

76  Fl  *  intarpx(El  .(x+d-vxl-D/dalta+l) ; 

77  out  ■  lsnan(Fl) ,  if  any(out()),  Fl(out)  ■  II (out);  and, 

78  F2  *  iatarpx(E2 ,(x-d-vx2-l )/dalta+l ) ; 

79  out  •  isnaa(F2) ;  if  any(out()),  F2(out)  »  I2(out),  and, 

80 

81  taral  »  (0.5/araa)asua(sua<(Fl-»l) . “2  ♦  (F2-I2) . ’2) ) ; 

82  tara2  *  (0  S/araa) alaabda*sua(sua(px  "2  ♦  py  “2  ♦  qx*2  ♦  qy  "2)); 

83  tara3  *  (0.5/araa)*au*sua(sua((pa-p)  “2  ♦  (q#-q)  "2)>. 

84  Xdisp(spriatf( ’Taras  Xl2. 5f  Xl2  Sf  Xl2  5f ’ ,tanal ,tara2 ,tara3) ) ; 

85  J  -  [t anal ♦tara2+tarm3, taral , tara2 , tar»3] , 

86 

87  if  nargout>l , 

88  Ftl  •  [B1;F1;11  abs(FX-tl)] ; 

89  FR2  •  [B2 ; F2 ; 12 ; abs(F2-I2) J ; 

90  and 

91 

B.1.3  hbdfss_cost2c.m 

1  function  tJ.m,m]»hbdfnn_cont(z,n,onr«u,El,E2 Jonbda) 

2  XHBDFSS_C0ST2C  Coot  function  for  depth  shoding  nnd  storoo 

3  I  problea  with  hoirnrchicnl  bunio  function*  Otoo  true 

4  I  pornpoctioo  project  ion 


a 


a 


a 


a 


a 


a 


a 


a 


a 


a 


a 


a 


•  a  a 
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5 

6 

8 

9 

10 
1 1 
12 

13 

14 

15 

16 
1' 
18 

19 

20 
21 
22 

23 

24 

25 

26 

28 

29 

30 

31 

32 

33 

34 

35 

36 
3" 

38 

39 

40 

41 

42 

43 
♦  4 

45 

46 


48 

49 

50 

51 

52 

53 

54 

55 

56 
5? 

58 

59 

60 
61 
62 

63 

64 

65 

66 
6" 
68 
69 


”2 

"3 

74 

75 

76 

7? 

78 

79 

80 
81 
82 
83 


X 

X  J*HBDFSS_C0ST2C(2 .1 .PARAHS , El ,E2 .LAMBDA)  shars  2  is  ths  surfacs 
X  hsight .  ■  is  ths  nuabsr  of  H-basis  lsvsls  El  and  E2  srs  ths  input 
X  zaagss.  LAHBDA  is  ths  scalar  ssighting  factor  on  dsparturs  froa 
X  saoothnsss  Ths  laags  paraastsrs  PARAHS  *  [f ,b,zO,psl ,qsl ,ps2 ,ps2] 

X 

X  J»  [Jtot .Jiaaga , Jsaooth] 

X  Clay  H  Thot^>son  2-24-91 

X  Rsvissd  4-3-91  by  csit 

X  Rsvissd  5-10-91  by  cat 

X  Rsvissd  5-19-91  by  cat 

X  Rsvissd  4-30-92  by  cat  to  support  aulti-grid  schsas 
X  Rsvissd  to  output  sstiaatsd  laags 

X  Caasra  constants 

f  *  paraasd ) ; 

b  »  paraasd) ; 

z0  *  parasisO)  . 

gaaaa  *  fsb/2/z0; 

dslta  *  paraas( 12 ) ; 

arsa  *  prod(*iz«(z> >*dslta*dslta . 

X  Light  Sourcs  positions 

psl  *  paraas<4) ,  qsl  *  paraas(5) ; 

ps2  *  paras«(6);  qs2  *  paraM(7); 

X  Caasra  coordinats  calibration 
vxl  *  paraas(8);  vyl  *  paraa*(9) ; 
vx2  ■  paraasdO);  vy2  *  paraasd  1)  ; 
vx  *  ( vxl*vx2)/2 .  vy  »  (vyl+vy2)/2. 

X  Spacial  coordxnatss  zn  zaag* 
f mz , nz]  *  sizs(z) • 

[x,y]  *  do«ain2d( [0:az-l3*dslta*vx+l , [0 : nz-lj •dslta+vysl ) ; 

X  Stancils 

hx  -  [-1  0  l]/(2*dslta) ;  X  X-dsrivativs 

hy  *  [1  ;0  ;-l]/(2*dslta)  ;  X  Y-dsnvativs 

hxx  *  [1  -2  1;  2-42,  1-2  l]/(4sdalta*dalta) ; 

Xhzy  *  [-1  0  1;0  0  0,1  0  -l]/(4*dslta*dslta> , 
hxy  *  [-1  1,1  -l] / (dslta*dslta) , 
hyy  *  Cl  2  1 ; -2  -4  -2;1  2  1]/ (4*dslta*dslta) ; 
dsl2  *  [1  4  1.4  -20  4; 1  4  l]/(6*dslta*dslta) ; 

X  Coaputs  p.q  using  nuasrzcal  dsnvativss  of  z. 
z  *  hbasis (z . n) ; 
zx  ■  cf iltsr2d(z .hx) . 
zy  ■  cf iltar2d(z ,hy) ; 
p  *  f*zx  /(x  *zx^z) ; 
q  *  fzy./iy  ♦zyv-x) ; 

X  Coaputs  rsflsctancs  asp  valuss 
Rl  *  raap(p ,q,psl ,qsl) , 

R2  *  raap(p ,q,ps2 .qs2) ; 

X  Coaputs  nuasrical  dsnvativss  of  z 
zzx  ■  cl iltsr2d(z, hxx) ; 
zxy  *  cf iltsr2d(z ,hxy ) . 
zyy  ■  cf iltsr2d(z ,hyy ) 

X  Coapots  disparity, 
d  ■  (fsb/2) . /z; 

X  Data rains  stsrso  aappsd  iaagss  Fl  ,F2  Sst  srror  to  zsro  vhsrs  F  is  lal 
Fl  »  intsrpxCEl , (x+d-vxl-t )/daltad ) , 

out  ■  isnan(Fl).  if  any (out ( : ) ) ,  Fl(out)  *  Rl(out),  snd,  ksspl  *  out; 

F2  »  intarpx(E2,(x-d-vx2-l)/dslta*l> ; 

out  ■  isnan(F2);  if  aay(out(:)),  F2<out)  »  R2<out);  and,  kasp2  *  out, 
tsral  •  (0  5/arsa)ssua(sua((Fl-Rl)  "2  ♦  (F2-R2)  *2) ) ; 

tsra2  *  (05/arsa)slas*da*(sua(sua(zxx  *2) )  ♦  2*sua(sua(zxy . '2) )  ♦  sua(sua(zyy  *2) ) ) . 
Xtsra2  *  (0  5/araa)«laabda«sua(sua(dalsqz  "2>>. 

Xdisp(spnntf  ( ’Taras  Xl2  5f  Xl2  5f  ’ , tsral  ,tsra2) ) ; 

J  a  [tsral*tsra2 ,  tsral ,tsra2] , 

if  nargout>l , 

Ftl  -  [El .Fl ; tl ; abs(Fl-Rl ) ; ksspl] . 

FR2  -  [E2 , F2 ; R2 , abs (F2-R2 ) ; kssp2] ; 


•  • 


•  • 


•  • 
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84  «ad 

B.1.4  hbdfss_grad2c.m 


1  function  a*hbdf aa_grad(z  ,n .paraaa  ,E1  ,£2  .lambda) 

2  XHBDF3S.GBAS2C  Gradient  function  for  dopth  fron  aheding  and  atoroo 

3  X  problen  oith  hoirarchical  bam  function!  Unoi  truo  perapective 

4  X  projection 

3  X 

6  X  G»HBDFSS_GBAD2C(Z ,1 , PABARS ,E1 ,E2 .LAMBDA)  there  Z  in  the  height 
-  X  Bap,  1  la  tha  nuBber  of  N-baaia  level!  El  and  E2  are  the  input 
s  X  imagea ,  LAMBDA  ia  the  acalar  aoighting  factor  on  departure  fron 
9  X  anoothneaa.  Tha  lnago  paranotara  PAAAMS  ■  [f .b.xO.pal ,qal ,pa2 ,pa2] : 

10 

11  X  Clay  M.  Thon paon  2-24-91 

12  X  Boviaod  4-3-91  by  emt 

13  X  Boviaod  5-10-91  by  cat. 

14  X  Boviaod  5-19-91  by  cat 

15  X  Boviaod  4-30-92  by  cat  for  multi-grid  acheae 

16 


X  Camera  conatanta 

f  •  paraaa ( 1 ) ; 

b  *  paraaa  ( 2 ) , 

zO  »  paraaa(3> , 

gaaaa  •  feb/2/xO; 

delta  ■  paraaa(12) ; 

area  •  prod(aiz*(z) >edeltaedelta; 

X  Light  Source  poaitiona 

pal  »  paraaa(4):  qal  “  paraaa(5); 

pa2  “  paraaa (6);  qa2  •  paraaa (7) ; 

X  Caaora  coordinate  calibration 
vxl  •  par nan (8 )  vyl  •  paraaa ( 9 ) ; 

vx2  •  paraaa(lO),  vy2  ■  paraaa(ll); 
vx  ■  (vxl*vx2>/2:  vy  «  (vyl*vy2)/2; 


X  Special  coordinate!  in  inaga. 

35  [az.nz]  •  aize(z), 

36  [x.y]  ■  doaain2d( tO:ax-l]adeltaevx*l , [0:nz-l]edolta*vy*l) ; 


37 


38  X  Stoncila 

39  hx  a  [-1  0  1] /2/delta:  X  X-derivative 

40  hy  «  [1  ;0  -l]/2/delta;  X  T-donvative 

41  hxx  -  [1  -2  1.  2  -4  2,  1  -2  l]/4/dolta/delta, 

42  Xhxy  -  [-1  0  1 ; 0  0  0:1  0  -1] /4/delta/dal ta 

43  hxy  e  [-1  i;i  -1] /dolta/delta , 

44  hyy  e  [l  j  i;-2  -4  -2:1  2  l]/4/delta/delta . 

45  del2  •  [1  4  1:4  -20  4:1  4  l]/6/delta/delta . 

46 


47  x  Compute  p.q  uaing  numerical  dorivativea  of  x 

48  z  •  hbaeia(z,n> : 

49  zx  a  cf ilt or2d(z ,hx ) , 
so  zy  *  cf ilter2d(z,hy) : 

51  p  *  fexx  /(x  ezx+z): 

52  q  a  f»zy  /(y.ezy+z) : 

53 

54  X  Coaputa  rof lectance  nap  value! 

55  B1  ■  raap(p,q,pal ,qal) ; 

56  B2  •  raap(p,q,pa2,qa2) , 

5'  Bpl  •  raapp(p,q,pal,qal) : 

58  Bp2  •  raapp(p,q,pa2,qa2) ; 

59  Bql  •  raapq<p,q,pel,qal) ; 

60  Bq2  •  raapq(p ,q ,pa2 ,q»2 ) , 

61 

62  X  Coaputo  nuaerlcal  dorivativea  of  z 

63  zxx  •  cf ilter2d(z ,hxx> . 

64  zxy  ■  cf iltor2d(x ,hxy) : 

65  zyy  •  cf ilter2d(x,hyy) : 

66 


6'  X  Coaputo  dieparity 

68  d  »  (feb/2) ./x: 

69  xl  »  (x*d-vxl-l)/doltaoi ; 

70  x2  •  (x-d-vx2-l )/dolta+l ; 

71 

72  X  Coyote  lat  difforoaco  ia  the  x  direction  of  iaago  arraya 

73  1  Baaed  on  the  ataacil:  -1  1 

74  lal ,nl]  ■  aizo(El).  Ca2,n2]  •  aixe(E2>. 


B.l.  COST  FUNCTION  AND  GRADIENT  ROUTINES 


1:57 


?s  stencil  ■  t“l  l]/delt*. 

76  Exl  ■  f ilter2d(El .stencil , ’ resize ’ ) , 

"  Ex2  ■  filter2d(E2 .stencil, 'resize' ) , 

rs 

79  X  Determine  stereo  mapped  image  derivatives 

so  Fxl  ■  interpx(Exl .floor (xl )) ;  zz  *  zeros (ml ,nl ) . 

si  out  *  lsnan(Fxl);  if  any(out(:)),  Fxl (out)  *  zz(out).  end 

82  Fx2  *  interpx(Ex2 ,floor(x2) ) ;  zz  *  zeros(m2 ,n2) ; 

83  out  *  isnnn(Fx2);  if  any(out(:)),  Fx2(out)  *  zz(out);  end 

84 

83  X  Determine  stereo  mapped  images  Fl,F2.  Set  error  to  zero  ehere  F  is  lal 
86  FI  *  mterpx(El  ,xl ) , 

8’  out  »  isnan(Fl);  if  any(out(:>),  Fl(out)  *  El(out);  end 

88  F2  *  interpx(E2 ,x2) ; 

89  out  *  isnan(F2) ,  if  any(out()),  F2(out)  *  R2(out).  end 

90 

91  X  Compute  Gradient 

92  G  *  zeros (mx ,nz) . 

93  EEpl  -  (Fl-El)  •Epl;  ERql  ■  (Fl-Rl)  *Rql ; 

94  ERp2  •  (F2-E2)  •Rp2.  ERq2  «  (F2-R2)  *Rq2 ; 

95  denx  *  (x  *zx+z)  “2; 

96  deny  *  (y  *zy*z)  “2; 

9-  G  *  -(f*b/2)*((Fl-Rl)  *Fxl  -  (F2-E2) . •Fx2)  /(*  *2)  ♦  ... 

98  f e((ERpl+£Rp2 )  *zx  /denx  ♦  (EEql*ERq2)  *zy  /deny ) . 

99  C(:)  ■  G  *  cf liter 2d ( (ERpl+ERp2 )  .•*. /denx, -f*hx, ’grad’), 

100  G( : )  »  G  •  cfilter2d((EEql*EEq2)  ex ./deny, -f*hy, ’grad’ ) ; 

101  G ( : )  *  G  ♦  cf ilter2d(zxx .lambdaehxx , ’grad * )  ♦  ... 

102  cf ilter2d(zxy ,2*lambda*hxy , ’grad’ )  ♦  ... 

103  cfilter2d(zyy .lambdaehyy , ’grad’ ) ; 

104  G  *  hbasis (G*(l /area) ,n , 'trams ’) ; 

B.l. 5  hbdfss_cost4.m 

1  function  [J.Ftl  .nL2]«hbdfaz_coat4(»  ,l.v«l«  .parama  .El  ,E2, lambda, mu) 

2  XHBDFSS.C0ST4  Coot  function  for  dapth  from  ahading  and  ataroo 

3  l  problaai  uaing  hiararchical  baaia  functiona  Uaaa  truo 

4  X  parapactiva  projaction  and  dual  z  upa. 

b  X 

6  X  J*HBDFSS_C0ST4 ( V ,  LEVELS ,  PAJUKS  ,  El ,  E2  .LAMBDA .  MU)  ahara 

•  X  V  •  [zl( : )  ;zJ( :  )J  ara  tha  optimization  aanablaa,  LEVELS  ia  tha 

8  X  numbar  of  h-baaia  lavala  El  and  E2  ara  tha  input  uaagaa ,  LAMBDA 

9  X  ia  tha  acalar  aaighting  factor  on  dapartura  from  aaoothnaaa  and  MU 

•o  X  ia  tha  acalar  aaighting  factor  on  atarao  matching  Tha  imago 

n  X  paraaatara  ara  PABARS  a  [f ,b ,z0 ,pal ,qil ,pa2 ,pa2] 

u  X 

13  X  J  «  [Jtot , Jiaaga , Jaaooth , Jatarao] 

14 

lb  X  Clay  K  Thoapaon  7-18-91 

16  X  Raviaad  to  aupport  aulti-grid  achaaa 

1' 

18  X  Caaara  conatanta 

19  f  •  paraaa(l) . 

20  b  ■  parama (2) . 

21  zO  •  paraaaO) . 

22  dalta  •  paraaa<12). 

23  araa  •  prod(aiza(El) )adaltaadalta. 

24 

2b  X  Light  Sourca  poaitiona 

26  pal  «  paraaaM) .  qal  -  paraaa(S). 

2'  pal  •  paraaa(B) .  qa2  -  paraaa(7) . 

28 

29  X  Caaara  coordinata  calibration 

10  vxl  a  paraaa(B),  ryl  •  paraaj(9), 

3:  vz2  •  paraaa(lO);  vy2  •  paraaa(ll); 

32 

33  X  Eztract  zl  *  z2  and  tranafora  into  nodal  baaia 

34  [azl.nzl]  •  aiza(El).  azl  •  nzl+2,  nzl  •  nzl*2 ; 

3b  [mz2,nz2J  •  aiza(E2) .  az2  •  az2+2 ,  nz2  •  nz2+2: 

36  zl  •  zaroa(azl .till .  *1 (>  •  »( 1  azl*nzi } , 

3-  z2  a  zaroa(az2,nz2) .  z2()  •  a(azl*nzl«[l  nz2*nz2) ) ; 

38  zl(  )  •  hbaaialil .lavala) ; 

39  z2()  ■  hbaaia(z2 , laaala) ; 

40 

41  X  Spacial  coordinataa  in  laaga 

42  Czl.yl]  »  doaain2d<  [0  aul-l>daltaa»zl*l  ,  [0  nzl-l]adalta*yyl  +  l )  ; 

43  [x2,y2]  *  doaain2d([O  azl-l]adalta*rz2*l , CO :nzl-l]*daltaaay2al ) , 

44 

4b  X  Stancila 

46  hz  •  [-1  0  l]/2/dalta.  X  I-darivatiaa 


e 


e 


e 


e 


e  e 


•  e  e  e 
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4  *  hy  *  [l;0;-l]/2/dslts.  X  Y-dsrivat  it# 

48  d«12  -  [1  4  1;4  -20  4;1  4  l]/6/dslts/dslta . 

49 

so  X  Coaput*  p,q  using  nuasncal  dsnvativss  of  z 

si  zx 1  ■  f ilt«r2d(xl ,Kx , ’rssizs 1 ) ;  zyl  *  f ilt«r2d(zl  .hy  .  ’rssizs  ’ ) ; 

52  zz2  «  filtsr2d(z2 ,hx , ’rssizs’ ) ;  zy2  *  filtsr2d(z2.hy , ’rssizs ’) . 

53  rows  ■  2  azl-l,  cols  *  2  nzl-1 ; 

54  pi  *  fszxl(ross,  >  /(xKrots. cols)  szxKross,  :  )*zl(ross, cols)) ; 
ss  ql  *  f*zyl(  ,cols>  /(yl Cross . cols)  *zyl( : , cols >♦*! ( ross , cols) ) ; 

56  ross  *  2az2-l.  cols  *  2:nz2-l, 

57  p2  *  fszx2(ross,:)/(x2(ross,cols)*zx2(ross.)4-z2(ross.cols)>. 

58  q2  *  f*zy2(  , cols)  /(y2(ross. cols)  *ry2(  , cols)+z2(ross ,cols) ) , 

59 

60  X  Coaputs  rsf lsctsncs  asp  vaiuss 

61  El  »  rasp (pi ,ql ,psl ,qsl ) , 

62  R2  «  rasp(p2,q2.ps2,qs2) , 

63 

64  X  Coaputs  intsrpolatsd  lasgss  Sst  srror  to  zsro  shsrs  zbsr  is  I si . 

65  zbsrl  *  lcubicCxlCl,  )’ ,zl * ,(x2*(f*b) ./z2> ' ) ’ ; 

66  out  *  isnsnCzbsrl ) ;  if  any (out< : ) ) ,  zbarl(out)  ■  z2(out);  and 
6?  zbar2  *  icubic(x2(l, : ) ’ ,z2 ’ ,(xl“(f»b)  /zl) ’ ) ' ; 

66  out  *  isnan(zbar2)  if  any(out(:)>,  zbar2(out)  *  zl(out),  and 

69 

to  tsral  *  (0  5/arss)*sua(sua((El-El)  “2  ♦  (E2-E2)  “2));  X  sfs 
-l  tsra2  *  (0  5/srsa)sls»bds*sua(sua( 

-2  filtsr2d(zl ,dsl2, ’rssizs * )  “2  ♦  f iltsr2d(z2 ,ds!2 , ’rssizs ’). ‘2  ));  X  Saooth 

'3  tsra3  *  (0  5/xraa)*au*sua(aua((zl-zbar2)  “2  ♦  (zbsrl-z2)  *2));  X  Stsrso 
74  XdispCsprmtf  ( ‘Taras  X12  Sf  Xl2  5f  Xl2  Sf  ’  .tsral  ,tsr*2 ,tsra3) )  ; 

-s  J  ■  [tsral^tsra2^tsra3, tsral .tsrai ,tsra3] ; 

76 

77  if  nargout>l , 

78  FEl  -  [El ;El , sbsCEl-Rl )] , 

79  FE2  *  [E2;E2;sbs(E2-E2)J  ; 
so  and 

B.1.6  hbdfss_grad4.m 

1  function  G>hbdfas_grad4(v, levels, paraas.El ,E2 , laabda.au> 

2  XKBDFSS.GBAD4  Gradient  function  for  depth  fron  shading  and  otoroo 

3  X  problon  using  hierarchical  bnain  representation  Uses  true 
«  l  perspective  projection  and  dual  z  asps. 

5  X 

6  X  G»HBDFSS_G*AD4(V  LEVELS,  PABAMS ,  El  ,E2,  LAMBDA,  MU)  Share 

:  X  V  •  [zl( : )  ;*2( : )]  are  the  optimization  variables,  LEVELS  is  the 

a  l  number  of  h-basis  levels  El  and  E2  are  the  input  images ,  LAMBDA 

9  X  is  the  scalar  weighting  factor  on  departure  froa  smoothness  and  MU 

13  X  is  the  scalar  weighting  factor  on  stereo  notching  The  laage 

1 1  l  parameters  are  PABAMS  ■  ff ,b,zO,psl ,qsl ,ps2,ps2] 

12 

I?  X  Clay  M  Thoapson  7-18-91 

14 

is  X  Caaera  constants 
i 6  f  ■  parens ( 1 > , 
i’  b  ■  parans(2) . 

18  zO  ■  paraasO) 
i?  delta  «  paraas(12), 

20  area  ■  prod(sixe(El ) >edelta*delta. 

21 

22  X  Light  Source  positions 

23  psl  >  parens (4) .  qsl  •  paraas(S), 

24  ps2  ■  paraasO) ;  qs2  •  pax ana (7) ; 

24 

26  X  Caaera  coordinate  calibration 
2?  vzl  •  paraas(B) .  vyi  •  paraasO); 

28  vx2  •  paraas(lO).  vy2  •  parasu(ll). 

29 

30  X  Extract  zl  t  z2  and  transform  into  nodal  basis 

31  [azl .nzl]  ■  eize(El);  azl  •  azl+2,  nzl  •  nxl*2, 

32  lnz2,nz2J  ■  size(E2) ;  ax2  •  az2+2;  nz2  •  nz2*2 , 

33  zl  •  zeros  (aul ,  nzl)  i  zl(>  ■  v(l  axlsnzl). 

34  z2  •  zeros (nz2, nx2 ) .  z2()  •  v(azl*nzl*[l :nz2snz2]) ; 

36  zl(  )  •  hbasisizl .levels) , 

36  z2()  •  hbasis(z2, levels) ; 

3' 

38  X  Special  coordinates  in  iaage 

39  Cxl.yl]  *  doaam2d([0  sul-ljsdelta*vzl*l  ,[0  nzl-lj«delta*vyl+l) ; 

40  [x2,y2]  »  donam2d([0  nxl-l]edslta»vx2*l ,  [0:nzl-l]*delta*vy2+l>  , 

41 

42  X  Stencils 

43  hz  •  [-1  0  1] /2/delta i  X  I-derivative 


B.  1 ,  COST  Fl’SCTIOS  ASD  GRA DIEST  RO UTISES 


1:1M 


44  hy  »  [l ;0.-lJ/2/dalta.  X  Y-dsrisatisa 

4b  dal2  *  [l  4  1,4  -20  4,1  4  1 ] /6/dalt a/dal ta .  X  Laplacian 

46 

4'  X  Coaputs  p,q  using  nusancil  dsnvscivss  of  z 

48  zxl  »  f  iltar2d(zl  ,hx ,  ’rssizs’ ) ,  zyl  *  filtsr2d(zl  ,hy ,  'rssizs  ’ ) . 

49  zx2  »  filtsr2d(z2 ,hx, ’rssizs’ ) ,  zy2  »  filtar2d(z2 ,hy , 'rssizs 1 ) . 
iC  ross  •  2azl-l  cols  »  2:811-1: 

51  pi  »  fszxKross. :)  ./(xKross,  coin)  szxKross,  :  >szl(ross.  cols))  . 

52  ql  *  fsxyl(  .cols)  / ( yl (ross , cols)  szyl(  .cola  KzKrosa  ,cols>  > , 

5<  ross  *  2az2-l.  cols  »  2:nz2-l. 

54  p2  *  f«zx2(ross  ,  )  ./(x2(ross  .cols)  szx2(ross,  >*z2(ro*s ,  cols) ) 

55  q2  •  f»zy2(  , cols) ./(y2(ross , cols)  «zy2(  ,cols)*z2(ross.cols>) . 

56 

5'  X  Coaputs  rsflsctsncs  asp  vsluss 

58  R1  •  raspCpl ,ql ,psl ,qsl ) . 

59  Rpl  »  raspplpl ,ql ,psl ,qsl ) .  Rql  *  raapq(pl ,ql ,psl ,qsl) , 

60  R2  *  rasp(p2 ,q2 ,ps2 ,qs2) . 

61  Rp2  •  raspp(p2.q2,ps2,qs2) .  Rq2  *  mspq(p2 ,q2 ,ps2 ,qs2) . 

62 

63  X  Coaputs  intsrpolstsd  laagss  xnd  rslstsd  tsras  Sst  grsdiant  to  zsro  shsrs  zbsr  is  Isl 

64  d  ■  (f sb)  /zl ; 

65  zbsr2  *  icubic(x2(l , : ) ’ ,z2 ’ ,<xl-d) ' ) ’ , 

66  out  «  find(isnsn(zbsr2) > , 

6*  zbsrx2  »  dcubicx(x2(l . .)’ ,z2 (xl-d) ’) Vdsltt , 

68  if  langth(out )>0,  zbsr2(out)  *  zl(out).  zbsrz2(out)  ■  Oszl(out),  and 

69  zbarz2  »  dcubicz(x2(l , : ) 1 ,z2 ’ . <xl-d> ’ , (zl-zbar2> ’ ) 1 : 

-o 

"1  d  *  (f sb)  /z2 : 

-2  zbsrl  •  lcubictxKl .  )  ’  ,zl  ’ , (x2*d>  ’ )  ’  , 

-3  out  S  f ind ( isnsn( zbsr 1 ) ) , 

"4  zbsrxl  •  dcubicxlxKl ..)’  ,zl  (x2*d)  ’)  ’/dolts . 

-5  if  langth(out)>0,  zbsrl(out)  «  z2(out).  zbsrxi(out)  «  0sz2(out).  and 
-e  zbsrzl  »  dcubicz(xl(l ,  ) ’ ,zl ’ ,(x2*d) ’ , (zbsrl-z2) ’ ) ’ . 

-8  X  Coaputs  arror  tsras 

'9 

80  X  las  fora  grsdiant 

si  ross  «  2:azl-li  cols  •  2  nzl-1. 

82  ERp  •  (El-Rl)  .Rpl,  EXq  -  (El-Rl)  sRqi, 

53  dsnx  •  (xKross, cols)  szxKross,  )sxliross ,cols) )  “2, 

84  dsny  •  (yl (ross .cols)  szyl( : , cols )«zl( ross .cols) )  *2 ; 

85  dJdzl  a  zsros(azl ,nzl) ; 

86  dldzKross.  )  •  -f iltsr2d(ERp  sxi (ross .cols)  /dsnx ,-f»hx) . 
a-  dJdzl(  .cols)  »  dJdzl(  ,cols)  - 

88  filtsr2d(Elq  szl(ross.cols)  /dsny , -f shy) : 

89  dJdzKross  .cols)  s  dJdzl(ross.cols)  ♦ 

90  f«(ERp  szxl(ross,  )  /dsnx  ♦  ERq  *zyl ( . , cols)  /dsny). 

91  dJdzl (  )  -  dJdzl  ♦ 

92  lsabdssfiltar2d(f iltsr2d(zl ,dal2, ’rssizs’ ) ,dsl2)  ♦ 

9)  aus(zl-zbsr2)  s(l-(f*b)szbsrx2  /(zl  ”2))  ♦ 

94  auszbsrzl. 

95 

96  ross  *  2:az2-l.  cols  »  2  nz2-l. 

9-  ERp  s  (E2-R2)  *Rp2 .  ERq  »  (E2-R2)  »Rq2 . 

98  dsnx  •  (x2(ross. cols)  «zx2 ( ross,  )sz2( ross. cols))  ‘2. 

99  dsny  *  (y2(ross,cols)  «zy2(  ,cols)sz2(ross ,cols) )  "2, 

100  dJdz2  •  zsros(az2.nz2) . 

101  dJdz2(ross.  )  •  -f iltsr2d(ERp . »z2( ross  .cols)  /dsnx ,-fahz)  . 

102  dJdz2( :  .cols)  •  dJdz2(  .cols)  - 

103  filtsr2d(ERq  sx2(ross,cols)  /dsny ,-fshy) . 

104  dJdz2(ross  ,cols)  ■  dldz2(ross .cols)  ♦ 

■05  f*(ERp  •zxKross .:). /dsnx  *  ERq  »zy2(  , cols)  /dsny), 

106  dJdz2(  )  »  dJdx2  ♦ 

1C  lsabdssf iltsr2d(f iltsr2d(z2 ,d«12 . 'rssizs ' ) ,dsl2)  -  ... 

108  Bu*zbxrx2  - 

::9  aus(xb«rl-x2)  •( (f»b )»zbarxl  /(z2  ”2 ) *1 ) . 

lie 

in  X  Rsturn  grsdiant  in  h-bssis  coordinstss 

112  dJdzl  •  hbausis(dJdzl*(l/srss)  .lsvsls , 'trsns  ' )  , 

U3  dJdz2  a  hbssis(dJdx2s(l/srss) .lsvsls , ’trsns ’) , 

U4  0  »  [dJdzl (.)  ;dJdz2( :)]  , 

1  1  5 

B.1.7  hbdfss_cost7.m 

1  function  [ J ,FR1 ,FR2]»hbdfss_cost (u ,n .psrsas ,E1 ,E2 .lsabds) 

2  XHBDFSS.C0ST7  Cost  function  for  dspth  froa  shading  and  stsrso 
i  X  problsa  sith  hairarchicnl  basis  functions  Usss  trus 

4  X  psrspsetivs  proysetion 

5  X 

6  X  J-HB0FSS.C0ST7  (0,1,  PARIES  . El  .E2.LA1BD*)  shsrs  U  is  ths  disparity. 


•  • 


•  • 


•  •  •  • 
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"  X  and  I  is  th«  nuabar  of  H-basis  lsvsls  El  and  E2  ars  tha  input 
s  X  iaag« s ,  LAMBDA  is  tbs  scalar  saighting  factor  on  dsparturs  fro* 

9  X  aaoothnass.  Ths  iU|i  paraaatars  P AMAHS  *  [f ,b  ,zO,psl ,qsl ,ps2 ,ps2] 

1°  X 

11  X  J*  [Jtot  ,  Jiaaga  ,  Jaaooth] 

12 

13  X  Clay  H .  Thompson  S-23-92 

M  X  Usas  u  ■  fb/2*  *  fb/2z_0  to  anhanca  tha  nuasrical  stability  of  ths  algorithm. 

15  X  Eavisad  to  support  aulti-gTid  schaaa 

16 

17  X  Camara  constants 

18  f  ■  paraasd )  ; 

19b*  paramo (2) ; 

20  zO  *  paraas(3) ; 

21  uO  *  f*b/2/z0. 

22  dalta  -  paraas(l2) ; 

23  araa  ■  prod(sisa(u) )*dalta*dalta . 

24 

2b  X  Light  Sourca  positions 

26  psl  ■  paraaa(4) ;  qsl  •  paraas(S); 

2T  ps2  *  parasa(d);  qs2  ■  paramo  (7) ; 

28 

29  X  Castors  coordinata  calibration 

30  vxl  *  para*s<8) ;  vyl  *  paraas(9) . 

31  vx2  *  paraas(lO);  vy2  *  paraas(ll); 

32  vx  •  (vxl+v*2)/2;  ry  •  <vyl**y2>/2; 

33 

34  X  S pacinl  coordinatos  in  inaga 

35  faz.nz]  •  siza(u); 

36  L*.y]  *  doaain2d(L0:a*-l]adalta^vx^l , [0:nz-l]adaltaavyai) ; 

3’ 

38  X  Stancils 

39  hx  ■  [-1  0  l]/(2*dalta) ;  X  X-darivativs 

40  by  *  LI  ;0;-l]/(2*dalta) ;  X  T-danvatiaa 

41  hxx  *  [1  -2  1;  2-42;  1-2  l]/(4*daltaadalta) . 

42  Xhxy  ■  [-1  0  1 ; 0  0  0 ; 1  0  -l]/(4*dalta«dalta J ; 

43  bxy  *  [-1  t;l  -l]/(dalta*dalta) ; 

44  hyy  *  Cl  2  l;-2  -4  -2:1  2  l]/(4*dalta*dalta) ; 

45  dal2  •  [1  4  1;4  -20  4; 1  4  l]/(6«dalta*dolta) ; 

46 

47  x  Coaputa  p.q  using  nuasrical  dorivati vas  of  u 

48  u  *  hbasis(u.n) . 

49  ux  ■  cf  il t ar 2d  ( u ,  hx )  ; 
so  uy  *  cf iltor2d(u ,hy> , 

51  p  •  f*ux . /(x  *ux-u-u0) ; 

52  q  *  f*uy  /(y . *uy-u-uO) ; 

53 

54  X  Coaputa  raf 1 octane a  aap  values 
65  Ml  *  rwap(p,q,psl ,qsl) ; 

56  K2  *  raap(p ,q ,ps2 ,qs2) ; 

57 

58  X  Coaputa  nuasrical  darivativas  of  u. 

59  uxx  *  cf iltar2d(u ,hxx) . 

60  uxy  ■  cf iltar2d(u , hxy ) , 

61  uyy  »  cfiltar2d(u,hyy) ; 

62 

63  X  Datamina  starao  aappad  laagas  P1.F2.  Sat  arror  to  zaro  shara  F  is  laf 

64  FI  ■  intsrpx(Kl , (x*u*uO-vxl-l  )/daltaM) ; 

65  out  *  ianan(Fl),  if  any(out()),  FI (out)  ■  El (out);  and, 

66  F2  ■  iatsrpx(K2  ,  (x-u-uO-vx2-l  )/dalta*l )  , 

6'  out  ■  ianan(F2) .  if  any(out(:)),  F2(out)  *  E2(out);  and, 

68 

69  XiJMga(  [Fl  ,F2  ;El  ,E2  ,  abs(Fl-El ) ,  abs(F2-E2>]  ,  [0  1]) 

70 

'i  taral  *  (O.S/oroo>a*iuo(aua((Fl-Bl)  *2  ♦  (F2-E2)  *2>). 

72  tara2  ■  (0  S/araa)«laabda«(sua(sum(uxx  ‘2) )  ♦  2*sua(sua(uxy  *2))  ♦  ata(sum(uyy  *2))). 

73  Xtara2  *  (0  S/araa)«lsabda«sua(sua(dalsqz  “2) ) , 

74  Xdisp(sprintf( 'Taras :  Xl25f  Xl2Sf’ .taral ,tara2)> ; 

75  J  *  [tsrml+tantf,  taral  ,tsrm2)  ; 

76 

77  if  nargout>l , 

78  FBI  *  [K1 ;Fl ; El ; abs (Fl-El )1 ; 

79  FB2  •  lK2 ; F2 ; E2 ; abs (F2-E2 ) j . 
so  and 

B.1.8  hbdfss_grad7.m 

1  function  G^bdf  ss_grad(u  ,n  .paraas  ,11  ,E2  .lambda) 

2  XXBDFSS  J3EAD7  Oradlant  function  for  dapth  froa  shading  and  storao 


i 


B.  1 .  COST  FUNCTION  AND  GRADIENT  ROUTINES 


X  problsa  with  hairaxchical  basin  function*  U»o»  truo  parspactiv* 

X  projection. 

X 

X  Qa|(8DFS5_GEAD7(U  .i.PAJUUS ,E1 ,E2 .LAHBDA)  ohoro  U  in  the  disparity 
X  anp ,  and  I  in  the  nuabar  of  H-bnoio  1***1*  El  ud  E2  *r*  tht  input 
x  iaagsa,  LAMBDA  i*  th*  *c*l*r  oaighting  factor  on  dapartur*  frea 
X  saoothn***  th*  iaag*  paraaator*  PABAM5  ■  £f ,b,*0,p*l ,q*l ,ps2 ,ps2] ; 

X  Clay  M  Thoapaon  5-23-92 

X  U*«*  u  »  fb/2x  -  fb/22.0  to  *nhanc*  th*  nuaorical  stability  of  tha  algoritha 
x  lavisad  to  support  aulti-grid  algoritha 

X  Caaora  constants 

f  •  paraas ( 1 > ; 

b  “  paraa*(2)  ; 

xO  *  paraas(3); 

uO  ■  f ab/2/zO ; 

dolt*  •  p*r*a*(12). 

area  •  prod(*ia*(u))*d*lta*d*lta; 

X  light  Sourc*  positions 

psl  •  p*r*a*(4) ;  qsl  *  paraas(S) ; 

p*2  •  paraas (61 ;  q*2  •  paraas (7 ) ; 

X  Caaora  coordinate  calibration 
vzl  •  paraas (8) ;  vyl  •  paraas(9) ; 
vz2  a  paraaa(lO);  *y2  •  paraas(ll). 

*x  »  (*xl+*x2)/2;  *y  ■  <*yl**y2>/2, 


X  Spacial  coordinatas  in  iaag*. 

faz.nx]  «  sizo(u): 

x.y]  •  doaain2d(  tO  az-l]ad*lta-»»x*l .  [0:**-lJ*daltaa*y»l> ; 

X  Stsncils 

hx  »  [-1  0  ll/2/dalta.  X  I-d*ri»ati*o 
hy  ■  [1 :0.-lJ/2/d*lta;  X  Y-d*ri»ati»* 
hx*  *  [1  -2  1;  2  -4  2;  1  -2  l]/4/d*lta/d*lta; 

Xhxy  a  [-1  0  1:0  0  0.1  0  -l]/4/dalta/d*lta; 
hxy  «  £-1  1,1  -l] /dslta/dalta ; 
hyy  ■  [12  1 ; -2  -4  -2:1  2  l]/4/d*lta/d*lta; 
d*12  «  [1  4  1:4  -20  4;1  4  l]/6/d#lta/d*lta: 

X  Coaputa  p,q  using  nuaarical  dorivati***  of  z 

u  «  hbasis(a.n) : 

uu  ■  u  *  uO: 

ux  a  Cfilt*r2d(u,hx) , 

uy  >  cfiltar2d(u.hy) : 

danx  •  (x  aux-uu) : 

dony  •  (y  auy-uu) : 

p  •  faux  /danx: 

q  »  fauy./dsny: 

X  Squared  danoauators  for  grsdisnt  tones  bolos 
danx (  )  a  danx . *danx , 
d*ny(  >  «  dony.adony; 

X  Coaputa  roflsctanc*  asp  aaluas 
El  ■  raapCp.q.psl ,q»l) ; 

E2  a  raap(p,q,pa2.qs2) , 

Epl  •  raapptp.q.psl.qsl) ; 

Ep2  ■  naapp(p,q,ps2,q*2) : 

Eql  •  raapq(p,q,psl,qsl) ; 

Eq2  ■  raapq(p.q.p*2,qs2) : 

X  Coaputa  nuaorical  dariaativa*  of  z 
uxx  a  cfiltar2d(u,hxx> ; 
uxy  ■  cf ilt*r2d(u ,hxy) , 
uyy  ■  cfilt*r2d(*,hyy) ; 

X  Coaputa  iaag*  projactions. 
xl  ■  (x»uu-*xl-l)/d*ltaal ; 

*2  *  (i-uu-**2-l)/d*lt*oi ; 

X  Coaputa  1st  iiffaronc*  in  th*  x  direction  of  iaag*  arrays. 

X  Based  on  th*  stoncil:  -1  1 

lal.nl]  •  sizo(El):  Ca2,n2]  •  siz*(E2), 

stancil  «  £-1  l]/d*lta: 

Exl  •  filt*r2d(El .staacil , 'rosiza ’ ) ; 

Ex 2  a  f iltsr2d(E2 , staacil , ’rasixo’ > . 
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S3  X  Ditirtlu  ■(««•  aappad  lu|<  darivativas. 

84  Fzl  »  mtarpz(Ezl ,  floor  (zl )) ;  zz  •  saroa(al ,nl> , 

S3  oat  ■  xanaa(Fzl):  if  any (oat ( : ) > .  Fzl(oat)  *  zz(out);  and 
ss  Fzl  •  intarpz(Ez2 .floor (z2) ) ;  zz  ■  zaroa(a2 ,n2) ; 

S3  out  >  lanan(Fz2>;  if  any(out(:)),  Fz2(oot)  •  zz(oat);  and 
ss 

S9  X  Dataraina  atarao  aappad  iaagas  F1.F2.  Sat  arrar  to  zaro  ahara  F  is  lal 
90  FI  ■  intarpz(El ,zi) ; 

9t  out  •  isnaa(Fl>:  if  any(out(:)),  Fl(out)  «  El  (out),  and 

92  F2  a  intarpz(E2 ,z2) ; 

93  out  ■  ianan(F2) ;  if  any(out(:)),  F2(out>  »  E2(out);  and 

94 

93  X  Coaputa  Gradiant 

96  0  •  zaroa(az.nz) ; 

97  EEpl  «  (FI -El)  *Epl:  EEql  •  (Fl-El) aEql ; 

98  EEp2  •  (F2-E2)  «Ep2;  EEq2  «  (F2-E2)  aEq2 : 

99  G  »  ((Fl-El)  »Fzl  -  (F2-E2)  aFz2> 

100  -  f  •(  (Dtpl+EEp2)  .  auz./danz  ♦  (EEql*EEq2 )  »uy  /dany)  ; 

101  G( : )  »  G  ♦  cfiltar2d((EEpl+EEp2)  ana. /danz,-f»hx , ’grad’) ; 

102  G(:>  a  G  ♦  cfiltar2d((EEql*EEq2)  auu./dany ,-f»hy , ’grad’) ; 

103  G( : )  •  G  ♦  Cfiltar2d(uzz,laabdaahzz. 'grad’)  ♦  ... 

104  cfiltar2d(uzy.2alaabdaahzy, ’grad’)  ♦  ... 

103  efiltar2d(uyy ,laabda*hyy , ’grad’ > ; 

106 

107  G  •  hbasis(Go(l/araa) ,n, 'trana ’ ) ; 

B.1.9  hbdfss_cost8b.m 

1  function  [1 ,FE1 ,  FZ2]*hbdf ss.cost (v .n.paraaa ,E1 ,E2 .laabda ,au) 

2  XKBDFSS.C0ST8B  Coat  function  for  dapth  froa  shading  and  atarao 

3  X  problaa  oath  hoirorchical  basis  functions.  Daoa  trua 

4  X  parapactiva  projaction  and  varying  albado 

6  X  J-HB0F3S.C0ST8E( T  .1 . PIEM3 . El , E2 . UHEO*  . KU )  ahara  *»(Z,EH0]  ora 

7  X  tha  optiaizatzon  variablaa  (haight  and  albado) ,  I  is  tha  Busbar  of 
s  X  H-basis  lavals.  El  and  E2  ora  tha  input  iaagas.  LAHBDA  and  RU  ara 
9  X  scalar  saighting  factors  on  dapartura  froa  saoothnass  for  hoight 

10  X  and  albado,  raspactivaly.  Tha  iaaga  parasMtars  PAEU5  a 

11  1  [f ,b.x0,psl,qsl.ps2.ps2] 

12  X 

13  X  1*  [Jtot , J iaaga, Jaaooth , Jalbadoj 

13  X  Clay  8  Thoapaon  6-28-92 

16 

l'  X  Caaara  Constanta 
is  f  ■  paraas(l) : 

19  b  a  paraas(2) . 

20  zO  ■  paraas(3) . 

21  gasaaa  »  f*b/2/z0- 

22  dalta  ■  paraas(12). 

23  araa  •  prod(siza(El> )*dalta*dalta; 

24 

23  X  Light  Sourca  positions 

26  psl  a  paraatt(4)  ;  qsl  ■  paraas(S)  . 

27  ps2  >  paroas(d) ;  qs2  •  par ana (7) ; 

28 

29  X  Caaara  coordinata  calibration 

30  vzl  •  paraas(8);  vyl  •  poransO) ; 

31  vz2  ■  paraas(lO);  vy2  a  paraou(ll); 

32  vz  •  (vzlOvz2)/2;  vy  »  (vyl*vy2)/2; 

33 

34  X  Spacial  coordinates  is  iaaga. 

33  [stz.ns]  a  siza(Il), 

36  z  »  v(:,l:nz)i  rho  •  v(  :  ,nz4’[l :  nz]  )  ; 

37  [z.y]  •  daaaia2d( [0  az-1 j*dalta*vz»l . C0  az-l3 *dalta*vy*l ) , 

38 

39  X  Stancils 

40  hz  •  [-1  0  l]/(2adalta) ;  X  Z-darivativa 

41  hy  a  [1 ,0;-lJ/(2«dalta) ;  X  T-darivativa 

42  hzz  ■  (1  -2  1;  2-42:  1-2  l]/(4*daltaodalta) , 

43  Xhzy  ■  [-1  O  1:0  0  0:1  O  -l]/(4*daltaadalta) : 

44  hzy  ■  [-1  1;J  -l}/(dalta*dalta) ; 

43  hyy  •  [1  2  l;-2  -4  -2:1  2  l3/(4odaltaodalta> ; 

46  dal2  ■  [1  4  1.4  -20  4:1  4  l]/(6adaltaadalta) ; 

47  hr  •  [0  0  0:0  1  0:0  0  0]  -  onas(3.3)/9, 

48 

49  X  Coaputa  p.q  using  naancal  darivatlvas  of  s. 
so  i  ■  hbasis(s.n):  rho  ■  hbasls(rhs.n) ; 


•  • 


•  •  • 
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si  zx  ■  cfilter3d(x,hx) ; 
s 2  xy  >  cfilter2d(x,hy) ; 

53  p  »  f»XX./<X. axX*X> 1 

54  q  ■  f»xy./(y  •XT*X) ; 

55 

56  X  Coxpata  reflectance  up  values. 

57  Kl  *  rho  *rmap(p,q,psl  ,qsl) ; 

56  12  •  rho  •rmap(p,q,ps2,qs2); 

59 

60  X  Cosputa  nujssrical  derivatives  ot  x. 

61  xxi  ■  cfilter2d(x ,hxx> ; 

62  xxy  ■  cfiltsr2d(x,hxy> ; 

63  xyy  *  cfiltsr2d(x ,hyy) ; 

6« 

65  X  Compute  disparity. 

66  d  »  (f«b/2)  /x; 

67 

66  X  Dataraina  starao  mapped  images  F1,F2.  Sat  error  to  xaro  share  F  is  I el 

69  FI  »  intarpx(El  .(xed-vxl-D/deltael) ; 

70  out  •  isaaa(Fl);  if  a»y(ont< : >) ,  Fl(eut)  -  11 (out);  and, 

71  F2  «  intarpx(K2,(x-d-vx2-l)/dalta*l) ; 

72  out  ■  isnaii(F2) ;  if  any(out(:)),  F2(out)  ■  12(out);  and, 

74  Ximage( [FI  ,F2  ill  ,12]  ,  CO  1].S) 

76  tarml  *  (O.S/araa)asum(siss((Fl”ll)  *2  ♦  (F2-12) .  "2> ) , 

77  term2  ■  (0 .5/araa)alambda*(sum(sua(xxx  *2) )  ♦  2*sum(sum(xxy  ~2)>  ♦  sum (sum (xyy  *2) ) ) , 
76  tarm3  *  (O.S/araa>amo*sum(sum(cf iltar2d(rho ,hr) . *2) ) ; 

79  Xtera2  •  (0. S/area)alm»bdeasum(sum(delsqx  ‘2) ) ; 

60  Xdisp(sprihtf( ’Terms:  Xl2.Sf  Xl2.Sf  X12.Sf ’ .terml ,tai»2,tar«3) ) ; 
si  J  •  [tei*l*tetm2*ter*3.  terml ,term2 ,term3]  ; 

82 

83  if  nargout>l. 

84  FBI  *  [El ; FI ;B1 ;abs(F2-B2)l ; 

85  FB2  »  [E2;F2;B2.abs<F2-B2)J; 

86  ond 

B.l.  10  hbdfss_grad8b.m 


1 

2 

3 

4 

5 

6 

8 

9 

10 

11 

12 

13 

H 

15 

16 
1? 
18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 


function  G*hbdf  ss_grad(v  yn.pnroM.El  .E2  .lombdo  .sra) 

XHBDFSSJ3BAD8B  Qradiont  function  for  dopth  from  shading  and  storoo 
X  probloa  with  hoirarchical  basis  functions.  Usos  truo  perspective 
X  projection  and  varying  alba do 
X 

X  G-IIBDFSS_OBADaB(¥.I,PAEAMS. El, E2, LAMBDA, MU)  short  V*[Z,M0]  art 

X  tho  optimization  variables  (hoight  and  albodo),  I  is  tht  nuabir  of 

X  H-basis  1 ovals.  El  and  E2  art  tho  input  images,  LAMBDA  and  MU  art 

X  scalar  soighting  factors  on  departure  from  smoothness  for  hoight 
X  and  albodo.  respectively.  Tho  image  parameters  PABAMS  * 

X  [f .b.zO.psl ,qsl ,ps2 ,ps2] . 

X  Clay  M.  Thompson  6-2S-92 

X  Camera  constants 

f  •  paraM(l) ; 

b  ■  params(2) ; 

zO  ■  paramsO) ; 

gamma  •  fob/2/z0; 

dolta  ■  paraM<12) 

aroa  *  pred(size(El) )odelta*delta, 

X  Light  Source  positions 

psl  ■  paramo (4) ;  qsl  *  paramo (5), 

ps2  *  parson (6);  qs2  *  paramo (7); 

X  Camera  coordinato  calibration 
vxi  ■  paras* (6)  ;  vyl  »  paramsO); 
vz2  *  params(lO);  vy2  ■  params(ll); 
v*  ■  (vzl+vx2)/2;  sy  »  (sylosy2>/2; 


X  Spacial  coordinatos  in  imago. 

[mz.az]  *  siso(Bl); 
s  »  v(.,l:na);  rho  *  v< : .nz*[l :n*3 > ; 

[z.y]  ■  demein2d(C0:ms-ljedalta+vz+l, [0:ns*tf«dolta+vy+l> ; 


X  Stone ils 

hi  *  [-1  0  l] /2/delta;  X  X-derivative 
hy  »  [l ;0;-lJ /2/delta;  X  Y -derivation 
hzz  ■  Cl  -2  1;  2-42;  1-2  iJ/4/delta/delta; 


# 


o  o 


OOO 


o  o 
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41  Xhxy  •  [-1  0  1 ;0  0  0 ; l  0  -l]/4/daltn/dalta , 

43  hxy  ■  [-1  1:1  -l]/dalta/dalta; 

43  hyy  •  [1  2  1;*2  -4  -2;1  2  U/4/dalta/dalta ; 

44  dal2  *  [1  4  1 .4  -20  4;1  4  l]/«/dalta/dalta; 

43  hr  •  [0  0  0;0  1  0;0  0  0]  -  oaaa(3,3)/9; 

46 

47  X  Conpata  p.q  using  nuaarical  darivativas  of  z. 

46  z  ■  hboois(z,n);  rho  »  hbasis(rho.n) ; 

49  zz  •  cf iltar2d(x,hx) ; 

30  zy  •  cf iltor2d(z,hy) i 
si  p  ■  f  *zx . / (z . *zx+z ) ; 

53  q  *  faxy./(y.azy*z); 

53 

34  l  Coapnta  raflactanca  anp  valaas. 

55  U1  *  noap(p  ,q  ,psl  ,qol )  ,  >1  *  rho.altl; 

56  U2  ■  mtp(p ,q ,pz2 ,q*2) ;  (2  ■  rho.*tt2; 

57  tpl  •  rho.«naapp(p,q,psl ,qsl) ; 

5«  lp2  -  rho.«raapp(p,q,pa2,qs2) ; 

59  tql  »  rho.ananpq(p,q,psl,qsl) ; 

60  tq2  •  rho  *raapq(p  ,q  ,p»2  ,qs2) ; 

61 

63  X  Coaput*  nuaoricnl  dariaatiaaa  of  z. 

63  zxz  *  cf iltor2d(z  ,hzx) ; 

64  zxy  *  cfiltor2d(z,hxy) ; 

65  zyy  »  cfiltor2d(z ,hyy ) , 

66 

67  X  Coapato  diapnrity. 

M  d  •  (fob/2) ./z; 

69  xl  »  (x*d-*xl-l)/dolt*H ; 

70  x2  »  (x-d-*x2-l)/dolt«ol ; 

71 

73  X  Coapato  1st  dlfforaaco  in  tha  x  diraction  of  laaga  arrays. 

73  X  Baaad  on  tha  stancil:  -1  1 

74  [al.nl]  ■  sixa(Il);  [a2,n2]  ■  siza(E2) ; 

75  stancil  •  [-1  l]/dalta; 

76  Exl  »  filtar2d(El .stancil, ’rasiza’) ; 

77  Ex2  •  filtar2d(E2, stancil. ’raaiza’) ; 

76 

79  X  Datanaina  atarao  aappad  inaga  darivativas. 
so  Fxl  •  intarpz(Kxl , floor (xl) ) ;  zz  •  xaros(al ,al) ; 

81  oat  ■  isnan(Fxl) ;  if  any (oat ( : ) ) ,  Fxl (out)  *  zz(oat) ;  and 
83  Fx2  ■  intarpx(Ex2 , floor (z2 ) ) ;  zx  a  zaros(a2 ,n2) , 

83  oat  a  isnan(Fx2) ;  if  any(oat(:)).  Fx2(out)  •  zz(oat);  and 

84 

83  X  Datanaina  starao  aappad  iaagas  Fl  ,F2 .  Sat  arror  to  zaro  ahara  F  is  lal 

86  Fl  •  intarpx(El ,xl) . 

87  oat  •  isnan(Fl);  if  any(oat(:)),  Fl(oat)  •  Rl(aat),  and 

88  F2  a  intarpx(E2 ,x2) ; 

89  oat  ■  isnan(F2>;  if  any(oat(:)).  F2(ont)  •  42 (out ) ,  and 

90 

91  X  Coapata  Oradiant 

93  Ftl  a  (Fl-tl) .  Ft2  •  (F2-t2> , 

93  Dpi  a  Ftl .  *tpl ;  Etql  ■  Ftl  atql . 

94  Etp2  «  FI2.*tp2;  Dq2  •  FT 2  *Iq2. 

93  danx  •  (x  azx+x) . "2; 

96  dany  ■  (y . azy*z)  "2; 

97  Ox  -  -(fab/2)a(FRl  aFxl  -  Ft2  vFx2>  /(x  "2>  ♦  ... 

98  fa((Etpl+Etp2>  *zx./daax  ♦  (Etql*Etq2)  •zy  /dany); 

99  0x( : )  ■  flz  -  cfiltar2d((IRpl*Ilp2>*z/danz.-fahx, 'grad’); 

100  Qz( : )  ■  Ox  -  cf  iltar2d((Hql*Hq2)  »z .  /dany  ,-fahy,  ’grad’ ) ; 

101  0z( : )  a  Os  a  cfiltar2d(zxx , laabdnahxx .  'grad ' )  ♦  ... 

103  cfiltar2d(sxy,2alaabdaahxy, ’grad' )  ♦  ... 

103  cflltar2d(syy , laabdaahyy , 'grad ’ ) ; 

104 

103  Orho  •  cf iltar2d(cfiltar2d(rho,hr)  ,aa*hr,  'grad')  -  Ftl.vttl  -  Ft2.*U2', 

106 

107  Oz  a  hbasis(Oz*(l/arsa) ,n, ’trans 1 ) . 

108  Orho  ■  hbasis(Orhoa(l/araa) ,n, ’trass’ > ; 

109  0  «  [Ox, Orho]; 

110 

B.1.11  hbsfs.cost.m 

l  fonction  [J .Ft] ahbsf s.cast (z . lavala .par ms ,I,1is bda) 

3  XtMFS  Cost  function  for  shapa  fro*  shading  asing  hiararchical 

3  X  basis  faactions.  lasad  on  orthographic  projaction. 

4  X 

3  X  JamFS.COST(Z,LmU.PAJUUIS,C,LtnDt)  ahara  Z  is  tha  dspth  anp. 

6  X  LEVELS  is  tha  ana bar  sf  h-basis  laaals.  I  is  tha  input  iaaga,  and 


•  •  • 


•  • 
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143 


*  X  LAMBDA  i(  tho  scal.tr  vaighting  factor  on  departure  fro* 

8  X  aaioothaaaa.  The  mage  paranetara  aro  PAKAKS  •  [f ,b ,z0 ,pa ,qa] . 

9 

10  X  Clay  N  Thonpaon  12-16-91 

11 

12  X  Centra  coaataata 
nf«  parane(l) ; 

Mb*  parana(2> ; 
is  zO  ■  paraaa(3) ; 

16  dalta  ■  parana(12), 

iT  araa  »  prod(aiza(z))*dolta*delta, 

18 

19  X  Light  Soarco  poaitiona 

20  pa  •  parana(4);  qa  *  paraaa(S), 

21 

22  X  Extract  xl  t  z2  and  tranafon  into  nodal  baaia. 

20  baz.nz]  *  aiza(E);  nz  a  az+2  nz  •  nz+2 ; 

24  z  ■  hbaaialx, lovala) ; 

25 

26  X  Stone ila 

27  hx  •  (f/zO)a£— 1  0  1} /2/dolt a .  X  X-derivative 
2»  hy  a  (f/z0)v[l;0;-lJ/2/dolta;  X  Y-dorivative 

29  del 2  a  Cl  4  1 ;4  -20  4;1  4  l]/6/delta/delta ,  X  Laplacitn 

30 

31  X  Conputo  p,q  uaing  nuaorical  darivativaa  of  z 

32  roaa  •  2:nx-l;  cola  •  2:nz-l; 

33  p  •  filtar2d(z,hx, ’roaizo’ ) ;  p  •  pirova,:); 

34  q  a  filtar2d(z,hy, ’roaizo’);  q  •  q(:,cola>; 

35 

36  X  Coaputo  rofloctanco  nap  valuta . 

37  t  a  raap(p,q,pa,qa) ; 

38 

39  taral  »  (O.S/araa)aaua(aun((E-t) . *2) ) ;  X  afa 

40  tara2  *  (0. S/area) *lanbda*aun(aun(  f ilter2d(z ,dal2 , ’roaizo ’). "2  )); 

41  Xdiap(aprintf(’Taraa:  X12. 5f  Xl2.5f  taral ,  tom2) ) , 

42  J  «  [toral+tora2, taral ,tor»2]  ; 

43 

44  if  nargout>l , 

45  FB  «  [E ;  B ;  aba  (E-l)  ]  ; 

46  and 


X  Saoethnoat 


B.l. 12  hbsfs-grad.m 


1  function  Oahbafa_grad(z, lovala, parana .E.laabda) 

2  XHBDFSS.0BAD4  Oradxont  function  for  ahapo  froa  ahading  uaing 

3  X  hierarchical  baaia  ropraaontation .  Baaod  on  orthographic 

4  X  projoction. 

5  X 

6  X  G«HBSFS_GBAD( Z, LEVELS .PABAHS ,E .LAMBDA)  uhara  Z  it  tho  dapth  aap, 

-  X  LEVELS  ia  tho  nun bar  of  h-baaia  lovala.  E  it  tho  input  iaaga,  and 

s  X  LAMBDA  ia  tha  acalar  vaighting  factor  on  daparturo  froa 

9  X  aaoothnaaa.  Tha  iaaga  paraaotora  aro  PABAMS  •  [f ,b ,z0 ,pa ,qa] 

10 

11  X  Clay  H.  Thoapaon  12-16-91 

12  X  Baviaod  to  aupport  aulti-grid  achoao , 

13 

14  X  Caaara  conatanta 

15  f  •  parana(t) ; 

16  b  •  paraaa(2) ; 

17  zO  ■  ptraaa (3) ; 

ia  dalta  •  parana (12), 

19  araa  •  prod(aixe(z) >*deltaodelta ; 

20 

21  X  Light  Sourca  poaitiona 

22  pa  •  paraaa(4>;  qa  ■  parent (S); 

23 

24  X  Extract  zl  B  x2  and  tranafon  into  nodal  baaia. 

25  [nz  ,nz]  ■  aiza(E) ;  nz  ■  nx4-2;  nz  ■  nz+2; 

26  z  •  hbaaiaU , lovala) , 

27 

28  X  Stoncila 

29  hx  »  (f/z0)o[-l  0  ll/2/dolta;  X  1-dorivativa 

30  hy  »  (f/z0)o[l ;0;-lJ/2/dolta;  X  V-dorivativo 

31  dol2  >[1  4  1,4  -20  4;1  4  l]/S/dalta/4alta;  X  Uplacian 

32 

33  X  Coaputo  p.q  uaing  anorical  dorlvativoa  of  z. 

34  root  ■  2:nz-l;  cola  »  2:nz-l; 

35  p  ■  filtor2d(z,hx, ’roaizo’);  p  ■  p<ro«a,;>; 

38  q  ■  filtor2d(z,hy, ’roaizo’) ;  q  ■  q(:,cola); 

37  dal2z  *  filtor2d(z,dol2, ’roaizo’); 

38 


•  •  • 
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39  X  Compute  rtlltctuct  up  values  . 

40  I  »  rmap(p,q,ps ,qs) ; 

41  Ip  •  rnapp(p,q,ps,qs) :  Iq  »  rmapq(p ,q,ps ,q») ; 

43 

43  X  Coaputu  vrror  tuna 

44 

45  X  lou  fan  gradient 

46  a  •  zeros(mz ,nz) ; 

47  G(rovs.)  ■  f ilter2d((E'I)  *Ip,hx) ; 

4>  0  ( :  ,  cols)  *  G(:,cols)  ♦  filter2d((E-I)  elq.hy) ; 

49  G(:)  «  a  *  Iambda>filter2d(del2z ,dsl2) ; 

50 

51  X  Istura  gradient  in  h-basrs  coordinates 

52  G  a  hbasis(Ge(l/araa) .levels, ’trans’) ; 

B.2  Support  routines. 


B.2.1  rmap.m 

1  function  Iernap(p,q, ps.qs) 

2  XIMP  teflectance  aap  calculation 

3  X  IaIIUP(P,Q)  coaputes  tbs  reflectance  aap  image  of  the  surface  aitk 

4  X  the  gradients  P  and  Q  (in  the  z  and  y  direction  respectively). 

5  X  P  and  Q  are  matrices  that  contain  tha  gradients  over  a  rectangular 

6  X  grid. 

7  X 

s  X  I»UUP(P,q,Ps  ,Qs)  uses  the  light  source  direction  (Ps.qs). 

9  X 

io  X  Currently  implements:  LssAertian  reflectance 

n 

12  if  narginaa2, 

13  ps  •  .1;  qa  a  .j;  X  Light  source  direction 

14  end 

15 

16  [n.m]  ■  size(p) ; 

17  I  a  iu(  ( l*ps*p+qs*q)  ./  eqrtd+p . *p*q. eq)  ./  sqrt(l*ps*ps*qseqs)  ... 

is  , zeros (n.m)  ), 

19 

B.2. 2  rmapp.m 

1  function  Iparmapp(p,q,ps ,qs) 

2  XVUPP  teflectance  aap  partial  derivative  calculation 

3  X  IpatlUPP(P,Q)  coaputes  the  reflectance  map  I  partial  derivative  of 

4  X  the  surface  with  gradients  P  and  Q  (in  the  z  and  y  direction 

5  X  respectively).  P  and  Q  are  statrices  that  contain  the  gradients 

6  X  over  a  rectangular  grid. 

7  X 

s  X  tp”t!UPP(P,Q. Ps.qs)  uses  the  light  source  direction  (Ps.qs). 

9  X 

10  X  Currently  implements:  Lambertian  reflectance 

11 

12  if  nargin*”2, 

13  ps  •  .1;  qs  u  .1;  X  Light  source  direction. 

M  end 

15 

16  [n.m]  •  size(p) ; 

17  d  *  ones(a.m)  ♦  pep  ♦  q  *q; 
it  e  ■  ones (n.m)  ♦  peep  ♦  qseq; 

19  Ip  e  (ps  -  e  e  p  ./  d  )  ./  sqrt(d)  ./  sqrt(ieps*ps*qaeqs> , 

20 

21  adz  »  f ind(rmap(p,q.ps.qa>*»0) ; 

22  if  leagth(adz)X),  Ip(odx)  •  zerosdength(ndz)  ,1)  ;  end 

23 

B.2. 3  rmapq.m 

1  function  Iqvrmapqip.q.ps ,qs> 

2  XUUPq  leflectance  nap  partial  derivative  calculation 

3  X  Iq-tlUPq(P.q)  computes  the  reflectance  nap  T  partial  derivative  of 

4  X  the  surface  oith  gradients  P  and  q  (la  the  z  and  y  direction 

5  x  respectively)  P  and  Q  are  matrices  that  contain  the  gradients 
s  X  over  a  rectangular  grid. 

r  X 
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>  X  Rq-RHAP(J(P,Q,Pe,Qa)  ua as  tha  light  source  direction  (Pa.Qa). 

$  i  • 

10  X  Currently  iapleaeate:  Laabertian  reflectance 

11 

12  if  nargia— 2, 

13  pa  «  .1;  qa  »  .1;  1  Light  aource  diraction. 

la  and 

13 

16  [n.a]  •  eixe(p) ; 

IT  d  »  onea(n,a)  ♦  pap  ♦  q»q; 
li  a  ■  oaaa(a,a>  ♦  pa*p  *  qe»q; 

19  Rq  ■  (qa  -  a  •  q  /  d>  ./  aqrt(d)  ./  aqrtd+paapa+qaaqa)  ;  # 

20 

21  adi  ■  find(raap(p,q,pa,qe)--0) ; 

22  if  laagth(adx)X),  Rq(ndx)  «  aaroadaagth(ndx) ,  1)  ;  and 

23 


B.2.4  conjgrad.m 

1  function  [x.OPTIOIS ,1 ,hiatory]-conjgrad(FUI .x.OPTIOIS, GLAD, . . 

2  Pl,P2,P3.P4,P5,Pe,P7,P8,P9,P10> 

3  XCOIJGRAD  Conjugate-gradient  opt  ini zat ion 

4  X 

3  X  X  •  C0IJGRAD( ’ FUI  ’ , XO . OPTI01S , ’ GRAD 1 )  fiada  tha  ainiaaa  of  tha 

6  X  function  'PUI ’  uith  gradient  ’GRAD’  uaiag  a  coo  jug at  a -gradient 

7  X  optiaixation  algoritha.  XO  in  an  initial  condition.  QPTIOIS  in  a 

a  X  aactor  that  containa  optional  inforaation  for  tha  optiaixar  (aaa 

9  X  FOPTIOIS) .  'FUI '  and  ’GRAD’  are  atringa  that  contain  the  naaaa  of 

10  X  tho  coat  function  and  gradient  H-filen,  reapectively .  'PUI '  ahould 

11  X  return  a  acalar  function  value ,  f-FUI(x)  ’GRAD’  ahould  return  tha 

12  X  gradient  vector  (df/dx) ,  g  •  GRAD(x) 

13  X  _ 

14  X  Up  to  tan  paraaMtara  can  bo  paaaed  to  ’FUI’  and  ’GRAD1  uaing  X  > 

15  X  C0IJGRAD( 'fun' .XO.OPTIOIS, 'grad' ,P1,P2, . . .)  ae  f-FUI(x,Pl,P2, . . . > 

16  X  and  g-GRAD(x,Pl,P2, . .  .). 

17  _ 

it  X  Iota:  FUICTIOI  value  ia  returned  in  OPTIQIS(R)  and  gradient  value 

19  X  ia  returned  in  OPTIOISdS).  0PTI0IS(19)-History  nave  rata. 

20 

21  X  Clay  R.  Thoapaon  2-4-91 

22 

23  tol  »  l.e-8;  X  Hiniaaa  aliened  alpha 

24 

25  arror(nargchh(4,14.nargin>) ; 

26 

27  X  Fora  call  atringa - 
26  par aaa  •  □  . 

29  for  n-S: nargia 

30  parana  •  [paraaa , ’ ,P' ,int2otr(n-4)] ; 

31  and 

32  if  -any(FUI<4«),  fcall  >  [FUI , ’ (x ’ , paraaa alaa  fcall  «  FUI;  and 

33  if  " any (GRAD<48) .  gcall  •  [GRAD, *(x ’ .paraaa ;  alaa  gcall  >  GRAD;  and 

34  if  'any(FUI<4B) , 

35  LPARAH  •  []  ; 

36  for  n«S ; nar gin , 

37  LPARAH  «  [LPARAH, ’ ,P’ ,int2atr(n-2)] ; 

36  and 

39  SEARCH  -  [*[evul(’'x»Pl*xeP2;”),  FUI,  >(x',  LPARAH,’)]’]; 

40  alaa 

41  SEARCH  •  ['[eval(”x-PiaxnP2;  ”> ,  '  ,FU»,’] ']  ; 

42  and 

43  linacall  »  [ 'leearcM SEARCH ,0,2*alpha,f ,GRAD,x,ph ' .paraaa ,’>’]; 

44 

43  nvara  •  length(x( : )) ; 

46  [ax, ax]  «  aiaa(x) ; 

47 

46  X  Initialize  paraaatera 
49  beta  »  0; 

so  f  ■  aval  (fcall);  X  Fuad  ion  value 

51  fold  •  f; 

52  gk  *  aval (gcall);  X  Gradient 

53  pk  >  zeroa(ax ,nx) ;  X  Search  diraction 
34  gkaora  ■  aona(gk(:)); 

55  OPT  IMS  •  foptiona(OPTIOIS) ; 

36  OPTIOIS(IO)  ■  OPTIMS(IO)  ♦  1; 

57  OFTIWSUt)  ■  OPIIOIS(ll)  ♦  1; 

36  it  0PTI0*S(l«)—O,  alpha-. 01;  alaa,  alpha  -  2aOPTIOIS(18) ;  and  X  Initial  guaaa 
39 

60  if  aize(gk)'-aize(x) ,  arrorCTha  aiza  of  tha  gradiant  and  x  don”t  Batch.’);  and 

61  if  OPTIOIS(1)>0. 
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62  di*p(  ’Fob  Evals - Valaa - Stepsixe - Gradient ■>; 

63  di«p< [spriatf ( 'XS . Of  X12  3g  X12.3g  * ,0PTI0*S(10) , f (1 ) ,0PTI0IS(18) ) , 

64  spriatf ( ’X123g  gkaon)]  >; 

6s  and 

66  if  OPTIOIS(19)>0  *  aargout>3, 

6r  history  •  [OPTIOHSUO)  ,f,GPTIOIS<10)  .gkaon]  ; 

66  and 

ro  vhila  OFTIOIS (10) <0PTIQ«S( 14) , 

?l  X  Ta*t  for  coavorgonco 

72  if  gknor»<am0IS(2>. 

73  OPTIOIS(fi)  *  f(l>:  0PTI0IS(15>  *  gtaoiw; 


74 

75 

76 

77 

78 

79 

80 
81 
82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 

95 

96 

97 


di»p( ’Gradient  criteria  aot.1),  brook 
and 

if  OPTIOIS(7)—0, 

pk  -  ~gk  ♦  baco*pk ;  X  Coapata  aaorch  direction 

also 


J* 

and 


X  Do  w  inexact  lino  search  to  dotoraiao,  alpha. 
GRID  »  gk(:)’epk(:>; 
if  GkAD>0 , 

pk  «  -gk;  GXAD  »  -gk(: )  ’»gk(:> ; 
if  OPTIOIS(1)>0.  diap( ’kodiroct  search.’), 

[ alpha. f ,n, hoc]  *  eval(linoeall) ;  X  Liao  anarch 


and , 


OPTIOIS(IO)  *  OPTIOHSUO)  ♦  a; 
OPTIOHS(IS)  *  alpha; 


*  *  x  ♦  alphaepk; 
if  alpha>tol, 

X  gkold  «  gk;  X  loodod  only  oith  Polak-Xibiare  Rothod 

gk  -  eval(gcall) ;  OPTIOIS(ll)  •  OPTIOHS(ll)  ♦  1; 


98 

99 
100 
101 
102 
100 
106 

105 

106 
103 
108 

109 

110 
111 
112 

113 

114 

115 
118 
113 
118 

119 

120 
121 
122 

123 

124 


Xbota  »  ((gk-gkold) >«gk)/gknon;  X  Polak-Iibioro  Rothod 
Xgknon  *  aorm(gfc( :  )  > ; 

bota  *  non»(gk(:  )) /gkaora;  X  Fletcher-Hooves  Rothod 

gkaono  •  betaegknon; 
also 

bat a  •  0; 

alpha  *  tol . 

gkaora  *  non(gk(:>>; 

if  OPTIO«S(1)>0,  dispCkasat  alpha  ') ,  and 
and 

if  OPTIOIS(1)>0, 

disp< [spriatf ( ’XS  Of  X12  3g  Xl2  3g  '  .OPTIOHSUO)  ,f<l)  ,0PTI0HSUB>> . 
apriatf  ( 'X12 . 3g  '  .gkaon)  .has] ) ; 
and 

if  OPTIOHS<19)>0  t  aargoot>3, 

if  OPTIOIS ( 10 ) >“hist ory (length (history ( :  ,1>>  ,m0PTI0gSU9) , 
history  »  [history ; [OPTIOHSUO) ,f ,0PTI0IS< 18) , gkaora]]  ; 
and 
and 

X  Tost  for  convergence  oa  relative  chaago  ia  F. 
if  abs(  (f old(  1 )  -f  ( 1 ) )  /  (f old  ( 1  >*sps ) )  <0PTI MS  (3 ) , 
onnn(l)  ■  f  (1 ) ;  OPTIOIS(IS)  •  gkaon; 


125  diap( ’halatiaa  faaction  chaaga  criteria  ■at.*),  braak 

1 26  and 

123  fold  »  f; 

128  and 

U9  OPTIMS(S)  •  f(l) ; 
no  OPTIOISCIS)  »  gkaon; 

131  1  *  f; 


B.2.5  lsearch.m 


1  faaction  [alpha.f .a.hos]  a  Uaarch(FDf .xO.alphal .fO.gO.Pl .P2.P3.P«,P5,P6,FT,P«.P9,P10.H1 ,Pl2> 

2  XLSIASCH  Iaoxact  liaa  search 

3  X 

4  X  [ALPHA, F.H]  ■  LSIAHCHOFOT’ ,X, ALPHA, P.O)  parfons  aa  inexact 

5  X  liaa  search  of  ths  1-D  faactioa  ’FM’.  ’FDI’  is  a  string  variable 

6  X  that  defines  the  nano  of  aa  R-fllo  faactioa  to  bo  minimised  The 

r  X  faactioa  shsald  rotan  a  scalar  valaa,  faFM(x)  X  is  ths  initial 

8  X  starting  point.  P  and  0  are  the  valaa  and  gradient  of  ths  faactioa 

9  X  at  ths  initial  point  X.  0  mast  be  asgativs  (in.,  dosahill).  ALPHA 


•  • 


•  •  • 


•  • 
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10  X  is  u  initi&l  gusss  for  ths  stsp  lsngth  (is.  XI  *  X  ♦  ALPHA) 

11  X  LSEAftCI  r st urns  ths  stsp  lsngth  ALPHA,  ths  function  vilus  F,  nnd 

12  X  ths  nusfcsr  of  function  sv&luations  f 

13  X 

14  X  Up  to  ton  sdditianal  par sm tors  can  bs  passsd  to  'fun’  using 

lb  X  [ALPHA, F. I]  -  LSKAHCHC 'FUI'.X, ALPHA. F,G, PI ,P2 , P3 ,  >  in  shich  cass 

16  X  ths  function  is  callsd  using  FUI(z ,P1 ,P2 ,P3 , . . . ) . 

17 

is  X  Clay  H.  Thoapson  4-11-91 

19 

20  shovStsps  *  0; 

21  shovPlot  *  0; 

?2 

23  tol  •  It-T;  X  Hiniana  tolarablo  rain*  for  alpha 

24 

23  X  Chock  for  valid  initial  condition. 

26  if  g0>0,  orror( ’Initial  (radiant  is  not  doanhill.’);  and 

27 

21  X  Fora  call  string 

29  parasM  *  ’(s’; 

30  for  n»6:nargin 

31  parasu  »  [paraau , > ,P’ , int2atr(n-8)] ; 

33  if  -any(FUI<4«) .  feall  »  [FUI .params, » ) ’] ;  also  fcall  *  FUI,  and 

34 

35  hoo  «  '  ' ; 

36  n  «  O, 

3»  if  shooStops,  dispo - Alpha - Function  »aluo>).  and 

39  if  shooPlot , 

40  hold  off,  plot([0  Soalphal] , [fO(l>  fO(l)*S»alphal«gO] , ’b- 1 ) ,  hold  on 

41  plot(0,f0(l), >bo’) , 

42  s»-alphal :alphal/10: Soalphal ; 

43  and 

44 

45  shilo  n<9 , 

46  X  Evaluato  function  at  x*alphal. 

47  x  •  xOaalphal , 

4«  fi  •  aval(fcall);  n  »  n+1 ; 

49  if  shooStops,  disp([’l:  ’  ,nua2str(alphal ) ,  >  ’  ,nusi2str(fl) ,  ’  ’.hoo]),  and 

50  if  shooPlot,  plotlalphal ,fl(l) , 'bo') ,  and 

31 

52  X  Fit  quadratic  function  to  ths  points  xO.xl.  f  »  a»x*2  *  box  ♦  c. 

53  c«  toil) ;  b  •  gO;  a  •  (f l(l)-boalphal-c)/(alphai*alphal) ; 

54 

55  if  shooPlot,  plat(s,polyval([a,b,c] ,s> , * r— ’ ) ,  and  X  Fittod  curvo 

56 

57  X  Dotaraint  naxt  jump  basod  on  tho  quadratic  fit. 

59  if  (a>ops) ,  X  lusip  foroard  but  not  too  much. 

59  alphaq  ■  -b/(2»n) ; 

60  if  (alphaq<Soalphal)  A  ( alphaq ' >alphal ) , 

61  alpha2  ■  alphaq; 

62  hoo  •  [hoo, 'quadratic  fit;  '] ; 

63  also 

64  alpha2  »  Soalphal ; 

65  hoo  •  [hoo,  'liaitad  juap .  ’]  ; 

66  and 

67 

69  alsaif  (acaps) ,  X  Option*  is  aaxiaisB,  jtap  foroard 

69  alpha2  ■  S*alphal ; 

70  hoo  ■  [hoo, 'Incroasa  stop  sisa  (q) ;  ']; 

71 

72  and 

73 

74  X  Evaluato  function  at  x*alpha2. 

75  x  »  xO  ♦  alpha!; 

76  f2  »  aval ( fcall ) ;  n  ■  n*l ; 

77  if  shooStops,  disp(['2:  ’ ,nra2str(alpha2) , ’  ' ,nu*2str(f2(l) ) , ’  ’ ,hoo] ) ,  and 

79  if  shooPlot,  plot(alpha2 ,f2(l) , 'x ’ ) ,  and  X  Chosan  point 

79 

so  X  Chock  for  adaquato  solution 

si  X  if  (f2(lX«iu(f0(l) , f 1(1 ) ) )  A  (alpha2<alphal) , 

92  X  alpha  •  alpka2; 

93  X  f  ■  f2; 

94  X  braak  X  loraal  axit 

93  X  and 

96 

97  X  Try  to  fit  a  cubic  function,  f  *  aax*3  ♦  b*x'2  ♦  c*x  ♦  d. 

99  d  •  f0( 1 > ;  e  ■  gO; 

99  alphaaat  ■  [alphal'3  alphal.*2;  alpha!. *3  alpha2 . *2] ; 

90  if  rcandlalphaaatXaps,  X  Cubic  solution  is  invalid,  ass  quadratic 

91  alpha  •  alpha2; 


s 


s 


s 


s  s 


s  s 
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f  ■  f2; 
r«t«n 


X  lomel  exit 


«b  -  elphe»et \ [f 1 ( 1 ) -ceelphel-d ;  f  2  d ) -eeelphe2-d] ; 
a  •  abd) .  b  ■  eb<2> ; 

if  shoePlot.  plot<s,poly»al([a,b,c.d] ,s> ,'*-’) ,  end  I  fitted  cures 

X  Check  descrininent 
del  *  beb-3»eee . 

if  (del>0)  A  ebe(e>>epe,  X  Soles  for  root! 

elphe3  •  (-besqrt(del) )/(3*a) ; 
eleeif  absUXeps  X  Function  looks  VERY  quadratic 
elph&3  •  elphe2; 

olpho3  «  inf;  X  Fie*  inealid  fit. 
end 

if  elpkn3<0,  X  Reduce  step  nine  end  etert  oeer. 

olphnl  *  nin ( elphel , elphe2 ) /2 ; 
boo  •  [hoe, ’Reduce  step  eixe  (c);  ’] ; 

if  shoeSteps,  diop([’3:  ’ ,nu2str(elphe3) , ’  ’,hoe]),  end 

eleeif  alphe3— inf,  X  Increeee  step  sixe  ud  stert  over, 
elphel  ■  S*smx( elphel, elphe2) ; 
hoe  »  [hoe, ’Inealid  cubic  fit;  ; 

if  shoeSteps,  disp<[’3  ’  ,nu2str(elphe3) ,  ’.hoej),  end 

***}  ( el ph a3- el phe2)/ elphel  <  .01,  X  Ixtre  fuction  eealuation  not  necessary 
13  •  inf; 

olse 

hoe  ■  [hoe.’Cnbic  fit;  ’] ; 

X  Realuate  fuction  et  x«alpha3. 
x  •  tO  *  elphe3; 

if  shoeSteps^disp([ ’3?  ’ lnu*2str(elpha3) , 1  ’  ,nu2str(f3(l>) ,  ’  >,hoe]),  end 

if  shoePlot,  plot (elphe3,f3(l) , ’x’ ) .  end  X  Chosen  point 
•nd 

X  Chock  if  miniau*  i*  brockotod. 
ftest  «  [fld>,f2d).f3d;]: 
l  -  find(nin(ftest>— ftest) ; 
elphe  -  e»el( [ ’elphe’ ,int2str(i(l))] ) ; 
f  «  eeelff’f ’,int2str(i(l))J); 

if  <f(lXfO(l))  »  (ebs(elphe)>tol) , 
return;  X  lonael  exit 

eleeif  ebs(elphe)<tol .  X  OAAD  nay  not  be  eccurete. 
x  »  xO  ♦  tol; 
f  •  evel(fcell);  n  ■  n*i; 
elphe  *  lOerudd  )*tol ; 
hoe  •  [hoe ,  ’kudos  Jusqi;  ; 
return 

else  , 

Xif  elphe— nin(  [elphel . elphe2 , elphe3J  ) , 
elphel  ■  elphe/2; 
hoe  •  [hoe.’Ieduce  step  sixe  <b>; 

X  else 

X  tipktl  ■  tlpkt; 

X  hoe  »  [hoe, ’Choose  snellest;  ’] ; 

end 


end  X  ehile 


X  Abnormal  exit,  lo  iaproeeaent  in  cost  use  m 
f  ■  fO; 

elphe  ■  0; 

dispCVUMO:  Exceeded  9  line  seerch  attupte.’) 


Use  nin  ealue  for  elphe 


B.2.6  fllter2d.m 


i  fuction  x  >  filtertdCe, stencil, resixe) 

1  XFILTER2D  Tee  diaeuionel  computational  stencil  fllterin*. 

4  X  X  *  FILTDt2DU,rn«CIL>  returns  X  ehich  is  the  result  of 
s  X  apply  is*  the  eaapetetionel  molecule  STUCIL  to  the  matrix  A. 
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6  t 

-  X  If  SIZS(A)  is  aa-by-na  and  SIZE(STEBCIL)  is  as-by-ns  than 
s  X  SIZE(X)  is  (aaeas-l)-by-(na»na-l> 

9  X 

10  X  FILTH 2DU.STEICIL)  is  tbs  saae  as  C0BV2(A .»0T9O(STEICIL,2> > 

11 

12  X  Clay  H  Thoapson  1-15-91 

13 

u  tas, nsl  »  size! stencil) ; 

13  laa.nej  ■  siza(a) ; 

16 

i-  X  This  calculation  is  the  saas  as  conv2(a,rot90(steacil,2>) 
u  X  but  it  faatar. 

19  if  0, 

20  x  ■  zerot(ms*aa-I ,nsena-l ) . 

21  for  i>l:as 

22  for  j«l:ns 

23  ■  »  stencil <as-i*l ,ns-j*l) ; 

2*  if  ■'■0. 

2i  x  »  x  ♦  [zeros(i-l ,  j-1) .zeroe<i-l,na> .zerot(i-l.ns-j) ; 

26  zeros(aa, j-1) ,oee,zero*(aa,a*-j) ; 

27  zeros (as- l , j-1 ) , zeros (as-i ,na) , zeros (as- 1 ,ns- j )]  , 

28  end 

29  end 

30  end 

31 

32  else 

33  X  ■  conv2(a,rot90(stencil,2>> ; 

34  and 

35 

36  if  nargin«*3 ,  X  heturn  the  central  (valid)  part . 

37  revs  «  as-1  ♦  fl :aa-«s*l] ; 

38  cols  »  ns-1  ♦  [1 :na-ns+l] ; 

39  x  •  x(ro«s  ,cols) ; 

40  end 

41 

B.2.7  cfilter2d.m 

1  function  e«cf iltar2d(a,b ,grad) 

2  XCFILTKB2D  Filtor  bicubic  approxiaat ion  of  array. 

3  X 

4  X  C  •  CFILTEX2D ( A , B )  applies  the  filter  B  to  the  array  A  shore 
3  X  A  is  interpolated  using  bicubic  interpolation. 

6  X 

7  x  C  •  CFILTER2D(A,B, ’grad’)  returns  the  convolution  of  B  sith  A 

8  X  for  a  gradient  calculation. 

9  X 

10  X  The  cubic  approximation  is  used  extrapolate  a  1  pixel  border 
n  X  around  the  array  A. 

12  X 

13  X  See  also:  FILTEB2D 

14 

15  X  Clay  K  Thompson  4-30-92 

16 

17  [aa.na]  ■  sixe(a);  X  Size  of  array. 

18  [ab.nb]  •  size(b) ,  X  Filter  size. 

19 

20  if  narginoa2, 

21  if  nb>I, 

22  a  •  [3oe(: ,l)-3oa( : ,2>*a( : ,3) ,a,3*a( : ,na)-3ea( : ,na-l)»e(: ,na-2>] ; 

23  end 

24  if  ab>l, 

23  a  ■  [3»a(l , : )-3ea(2, : )*a(3, : ) ;a;3»a(aa, : )-3ea(aa-l , : >*a(aa-2, : )] ; 

26  end 

27  co  filtar2d(a,b, ’resize’) ; 

28  else 

29  c  ■  f ilter2d(a,b) ; 

30  if  ab>l , 

31  c(..2:4)  •  c(:,24)  ♦  c(:,l)»t3  -3  1] ; 

32  c( :  ,na*nb-4:naenb-2)  •  e( : ,nnenb-4:ne*nb-2)  ♦  e< : ,na*ub-l)*[l  -3  3]; 

33  c  o  c<  :  ,  2  :nb*na-2) ; 

34  end 

33  if  ab>l , 

3«  c(2.4,:>  o  e(2.4,.>  ♦  [3; -3; l]ee(l , : ) ; 

37  c(ae*nb-4  :aa*ab-2 , : )  ■  c(aa*ab-4  :aa*ab-2 ,  : )  ♦  [l  ;-3,3]ec(aa*mb-l . : ) ; 

38  c  «  c(2:ab*ae-2, : ) ; 

3»  end 

40  end 


•  • 
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T- 


* 


B.2.8  hbasis.m 


1  function  y  »  hbnoioCx.n.opor) 

2  XMBASIS  Hop  fron  hiororchieol  bonio  roprooontntion  to  nodol 

3  X  ropranoatotion. 

s  x  Y  ■  KBASIS(X.B)  or  Y  «  HBASIS(X,«, ’nodal’ >  nopo  I  fron  a 
6  X  hierarchical  booio  oith  I  leaelo  to  tbo  nodol  booio 
-  ¥ 

»  x  Y  •  K»ASIS(X,1, ’trono’ )  nopo  fro*  tho  nodol  booio  to  tha 
,  X  biororcbicol  booio  oith  ■  looolo  uoing  tho  od joint  nop. 

a  x  Y  »  HBASISd  ,1 ,  ’in*  * )  nopo  fron  tho  nodol  booio  to  tho 

12  X  hiororchieol  booio  oith  I  looalo. 

a  X  Y  »  HBASISd, I,  ’tint’)  nopo  fron  tho  nodol  booio  to  tho 

14  x  hiororchieol  booio  oith  B  lovalo  uoing  tho  odjoiat  nop. 

!'  *  Baforonca:  "Foot  Sorfoco  Intarpolotion  Uoing  Hoirorchicol  Booio 
i»  X  Funetiono",  Bichord  Soalioki,  IEEE  PAHI.  Vol  12,  Bo.  6,  Juno  1990. 

20  X  Cloy  *  Thonpoon  4-2-91 

21  X  Boviood  6-10-91  by  CM 

23  ... 

23  orror(nurgchk(2 ,3 .norgia) ) ; 

24  if  norginO,  coda  »  ’no’;  oloo  coda  »  opor(l:2);  and 


2T  X  Chock  to  noka  aura  tha  oioa  of  *  io  eonpatibla  oith  n  lovalo. 


[an. an]  *  aixa(x) ; 
incr  ■  2*<n-l); 


30  1^orror?[^Eoch>>dinaaoian  of  X  nant  bo  lorgar  than  ’  ,int2otr(incr) ,  ... 

’  for  ’ ,int2otr(n) . ’  lovalo. ’]); 
and 

if  eodo^’no’,  X  opor*“ ’nodal ' 
y  a  xaroa(nB.nn) ; 

rooo  «  1  liner  inn-,  coin  «  1  liner  inn; 
y(rooo.colo)  «  x(rooo .cola) ; 

*°xrooo"«  2araa(aa-i liner)  >»  incr;  xcolo  •  2aran<nn-l . incr)  >«  incr; 
incr  *  incr/2; 

roo2  «  lincr:ni;  col2  a  1  incr:n>; 

y(roo2  coI2)  »  kb<y(rooo.colo> .xrooo. xcolo)  ♦  X(roo2,col2) , 
y(rooo.colo)  ■  y(rooo.colo)  -  x(rooo .cola) ,  X  Bonooa  oxtro  tom 
rooo  *  roo2:  cola  »  col2; 
and 


47 

40 

49 

40 

41 

42 

43 

44 
44 

46 

47 
40 
49 
60 
61 
62 

63 

64 
64 
66 
67 
66 

69 

70 

71 

72 

73 

74 
74 

76 

77 

70 

79 

00 

01 


aloaif  coda-n’tr’.  X  opor—’trano' . 
y  •  zaroo(an.nn) , 
incr  ■  1 ; 

rooo  »  lineman;  cola  ■  1  liner  inn; 
y(rooo.cola)  *  x(rooo ,colo) ; 
for  lnvnl»l : (n-1) , 
incr  *  2*incr; 

roo2  *  1 : incr: no;  eol2  »  liincnnn; 
y( roo2  ,col2>  ■  y(roo2,col2>  ♦  hbt<y(rooo,colo)) : 
rooo  *  roo2;  cola  *  col2; 
and 

aloaif  codo-n'in’,  X  opor«“  ’  inv  ’ , 
incr  ■  1 ; 

y  ■  xorao(an.nn) ; 

rooo  ■  1  liner  inn;  colo  ■  l:incr:nn; 
y(rooa.colo)  -  x(rooo.colo) ; 
for  lavol"l : (n-1) , 
incr  »  incr»2 , 

roo2  •  liincnnn;  eol2  »  l:iner:nn; 

xrooo  ■  2«r«<nn-l.iner)  >*  incr  .  xcolo  ■  2*ran(nn-l  .incr)  >■  mcr; 
y(rooo.colo)  »  y(rooa.cola)  -  hb(y(roo2 ,col2) .xrooo, xcolo) . 
y(roo2 ,col2)  »  r(roo2,col2>  ♦  x(roo2,col2) ;  X  Add  back  oxtro  torn 
rooo  •  roo2;  i»lo  ■  col2; 

ond 

aloaif  codo—'ti’,  X  opomn’tinv’, 
y  »  xarofl(an.na) ; 

root  ■  1 :incr inn;  colo  ■  1  liner  inn; 
y  »  x; 

for  looalo(n-l) : —1 : 1 . 
incr  *  incr/2; 

roo2  «  1 1  incr  co!2  »  lincrna; 
y(rooo.eolo)  ■  y(rooo.eolo)  -  hbt(y<roo2,col2)> ; 


•) 


* 
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rr 


* 


62  rou  ■  ro»2;  col#  ■  col 2; 

S3  ond 

M 

85  «1#0 

86  •rror([’Th*  oporotion  ’  ,opor,’  is  invalid  ’]>; 

87  «nd 

88 
89 


O 


■W 


+  : 


B.2.9  hb.m 

1  function  yahb(x,xtraros* .xtrncols) 

2  XKB  Intorpolnto  •  2-0  function  on  4  rectangular  grid  using 
1  X  Hierarchical  Basis  function*. 

4  X 

5  X  Y  ■  HB(X>  roturn*  X  interpolated  to  th*  next  level.  If  X  is 

6  X  a-by-n  thsn  Y  is  (2*a-l )-by-<2»n-l ) . 

6  X  Cloy  H.  Thompson  4-2-91 

9 

10  colaask  •  [5  5] , 

1 1  rotouk  ■  L  5  ;  .  SJ  ; 

12 

12  if  nargin<2,  xtraross  »  0;  ond 

14  if  norginO,  xtrncols  •  0;  and 

15 

10  [a,n]  •  sizo(x) ; 

it  if  xtrnrous  I  xtrncols, 

18  x  »  £x,xeros(a,xtraco'  z*ros(xtrnro»s ,n) ,x*ros(xtrnross , xtrncols)] , 

19  ond 

20 

21  [a,n]  *  sizs(x) , 

22  am  «  2*a-l;  no  »  2*n-l; 

23  oddross  *  l:2:na; 

24  nvnnrous  *  2:2:*«-l; 

25  oddcols  •  M  il 

26  ovoncols  •  2:2:nn-l; 

2' 

26  y  *  zsrosias.nn) . 

29  y( oddross .oddcols)  »  x . 

30  y(oddross,*u*ncals)  »  f ilter2d(x ,col-nak , ’rosizo’ ) ; 

31  y(evanross , : )  •  filtsr2d(y (oddross,  .rosansk , ’rosizo ’) ; 

32 

33  if  xtrnross  I  xtrncols, 

34  y  *  y<l :as-xtraross,l  nn-xtrncols) ; 

33  and 

B.2.10  hbt.m 

1  function  y  ■  hbt(x) 

2  XHBT  Dociaato  n  2-D  function  on  n  roctangular  grid  using  lisrnrchicnl  Basis 

3  X  functions. 

4  X 

3  X  Y  ■  HBT(X)  returns  X  dociantod  to  tha  next  conrsast  lasal. 

6  X  If  X  is  a-by-n  than  Y  is  <(a*l)/2)-by-((nel)/2> 

1  X 

8  X  Iota  (urU )  and  (n*l)  aust  ba  divisible  by  2. 

9 

10  X  Clay  H.  Thoapson  4-2-91 

11 

12  [a,n]  •  siza(x), 

13  ross  •  [l:2:u];  cols  •  fl2:n], 

14  a  ■  [0.zaro*(l,n) ,0;zaros(a,l) ,x,xaros(a,l) ;0,xaros(l ,n) ,0] ; 

13 

16  y  «  a(ro*s,  cols)  ♦  2a*(ross,  col**l)  ♦  stress ,  cols*2>  ♦  ... 

it  2*a(ro*s+l,cels)  ♦  2aa(ross*i ,cols*2)  ♦  ... 

18  a(ross*2,cols)  ♦  2aa(ross+2,cols*l)  ♦  »(ros**2  ,cols+2) ; 

19  y  ■  y/4 ; 

20 
21 

B.2.11  interpx.m 

1  function  F»int*rpx(E,x> 

2  X IITXkP I  Linear  interpolation  in  tha  x-diraction. 

•  X  Fa  irand.X)  returns  a  aatrix  F  containing  the  values  of  C 

5  X  at  the  points  X.  Tha  aatrix  X  asst  havs  the  a  mm  nuaber  of  ross  as 

6  X  I.  Th*  valaos  in  th*  aatrix  X  asst  bo  botsooa  1  and  I,  shore  B  is 

7  X  th*  auabor  of  coluaas  of  I.  Th*  vala*  Bal  sill  b*  returned  shoro 

8  X  this  is  not  th*  cas* 

9 


•  • 


4 
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10  X  Clay  W.  Tfcoapton  10- 17-90 

11 

12  t«,a]  ■  aixa(E) ; 

13  [u  ,nx]  *  ii  fit) ; 

M  if  trrorCZ  au«t  htv«  th*  iim  nu»b*r  of  rova  aa  E  .  ’ )  ;  and 

15 

16  X  Coaputo  nearest  x  position,  xloa. 

17  slow  »  floor (x) ; 
is  oot  ■  (x*»n> ; 

19  nout  *  aoai(out(  : ) )  ; 

20  if  aoat>0,  slonioot)  *  (n-l>*ones(nout  ,1  > .  end 

21 

22  X  Chock  for  oat  of  rone#  values  of  x  and  aot  to  1 

23  out  *  <x<l)|<x>a); 

24  noat  a  aoa(oat(:>); 

25  if  noat>0,  x(oat)  ■  onoa(noat ,1 ) ,  xlov(out)  ■  onoaCnoat , 1 ) ,  and 

26 

27  X  Determine  indox  into  matrix  elements 

28  X  Iota:  y*[l :m] *emones(l .ax) ; 

29  alaa  ■  [1 :*J  ’omonesd nx)  ♦  (xlov-l)*a; 

30  F  •  E(alaa)  ♦  (x-xlov)  . •  <E(alaa+a)-E(aloa) ) ; 

31 

32  X  Sat  valuaa  of  F  vhara  x  la  out  of  range  to  lal 

33  if  aay(out(:)),  F(oat)  ■  lal*ones(nout ,1 ) ;  and 

34 

B.2.12  domain2d.m 

1  function  [x,y]  *  domain2d(x ,y) 

2  XD0RAII2D  Generate  Z  ud  T  arrays  for  3-d  plot! . 

3  X  [XX, YT]  »  D0RAII2D(X ,T>  trail* fora*  tk*  domain  specified  by  vector* 

4  X  X  aad  Y  into  array*  XX  aad  YY  that  caa  ba  nsad  for  tha  avaluat ton 

s  X  of  foactioas  af  taa  variables  with  3-d  tMah  plot*.  For  example, 

el  to  avaluat*  th*  function  x*exp(-x"2-y*2>  ovar  the  rang*  -2  <  x  <  2 
-  X  -2  <  y  <  2, 

s  X 

9  X  [x,y]  *  meshdom(-2: .2:2,  -2  .2:2); 

10  X  x  *  x  •  axp(-x.*2  -  y  *2) ; 

11  X  noah(z) 

12 

13  X  J.I  Littla  12-2-86 

14  X  kaviaad  20-Ray-90,  LS. 

is  X  Copyright  (c)  1986,  1988,  1990  by  th*  RathMorks .  lac. 

16 

!'  nx  »  langth(x) ; 

H  ny  *  langth(y) ; 

19  x  »  x( : )  ’  X  nek*  aura  I  1*  a  ro*  vector 

20  x  >  x(on**(ny,  1) ,  ) . 

21  y  ■  y(ny : -1 : 1 ) ;  y  •  y< : ) ;  X  aaka  aura  y  i*  a  column  vector 

22  y  "  y(  ,onos(l ,  nx) ) : 

23 

B.2.13  icubic.m 

1  function  F»icubic(x.y ,u) 

2  XIOTBIC  Cubic  Intarpolation  of  a  l-D  function 

3  X 

4  X  F*IOTBIC(Y , XI )  return*  tha  value  of  th*  1-0  function  Y  at  th* 

3  X  point*  XI  uaing  cubic  intarpolation  i«agth(F)*l*agth(Il )  XI  l* 

6  X  an  index  into  th*  vector  Y  Y  l*  th*  value  of  th*  function 

'  X  avaluatad  uniformly  on  a  interval  If  Y  i*  a  matrix,  than 

e  X  the  interpolation  is  performed  for  each  column  of  Y 

9  X 

10  X  If  Y  i*  of  length  I  than  II  must  contain  value*  bataaan  1  and  f 

11  X  Th*  vain*  lal  i*  returned  if 

12  X  this  is  not  th*  case 

13  X 

14  X  F  *  ICTBICU.Y.Xl)  u»*»  th*  vector  X  to  specify  th*  coordinate* 

13  X  of  th*  underlying  interval  X  must  be  squally  spaced  and 

le  X  monotonic  XI  must  lie  sithin  th*  coordinate*  in  X 

i"  X 

it  X  See  also  ILIIEAI 

19 

20  X  Clay  R  Thompson  7-4-91 

21 

22  X  Based  on  "Cubic  Convolution  Interpolation  for  Digital  Image 

23  X  Processing",  Bobert  G  leys,  IBS  Tran*  on  Acoustics,  Speech,  and 

2*  X  Signal  Procoesing,  Vol  29,  Io .  6,  Dec.  1981,  pp.  1163-1180 

23 

26  if  nargin»2.  X  Io  X  specified 


4 


4 


4 


4 


4 
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2*  u  »  jr;  y  ■  z; 

2 i  X  Chack  for  vuctar  problam  If  oo,  mnka  avarything  a  column  vector 
29  if  min(»iza(y))m»l ,  y  •  y(:).  and 

20  if  min!aiza!u)>ami  ,  u  •  a(:>;  and 

21  [nrooa.ncola]  •  aixa(y); 

22 

22  alaaif  nargin— 3 ,  X  X  opacified 

24  X  Check  for  vector  problem.  If  eo,  make  avarything  a  column  vector 

25  if  niB(oiBe(y>)“ml,  y  «  y(:>;  end 

26  if  miB<eixe(z> )“1 ,  z  *  x(:);  ead 

27  if  aiiB(aize(u))m>l,  o  *  u(:>;  end 

is  [nrooe .ncole]  ■  eize(y); 

29  X  Scale  and  ahift  u  to  be  indicae  into  Y 

40  if  (min(oize(x))  ~*1) ,  arrorCX  moot  be  a  vector.’);  and 

41  I  »  X< : ) ; 

42  [b,b]  ■  eize(x) ; 

42  if  a  "«  nrooe , 

44  error! 'The  length  of  X  aunt  natch  tha  number  of  rone  of  Y.’>; 

45  end 

46  u  ■  1  ♦  <u-xCl))e((nrooe-l)/(x(a)-x(l))) ; 

47 

46  aloe 

49  error! ’Wrong  number  of  input  argumenta  ’ > ; 

50  end 

52  if  nroox<3,  error! ’Y  nuet  have  at  leaat  3  room.’);  end 
52  [n,n]  «  xize!u) ;  ,  ,  .  .  ,  .  «  _ 

54  if  n«*l,  u  •  u*onea!l , ncole) ;  [n,n]  •  aize(u) ;  end  X  Expand  u 

55  if  n'mncole,  error!’The  number  of  coluame  in  XI  and  Y  moat  match.');  end 

56 

57  X  Check  for  out  of  range  valuee  of  u  and  eat  to  1 

58  uout  a  find!!u<l) I luinrooe)) ; 

59  nuout  «  length! uout). 

60  if  nuoutX),  uiuout)  *  onee  1  nuout  .1 ) ;  end 

61 

62  X  Interpolation  paranatara 
12  a  •  !u  -  floor!u)); 

64  u  ■  floor!u) ;  ... 

65  d  •  f ind(u“nro»e> ;  if  lengthldjiO,  u!d)  ■  u(d)-l;  a(d)  •  e!d)  +  l;  end 

66 

67  X  Expand  y  ao  interpolation  ia  valid  at  the  boundary 

68  y  m  [3ey!l . ; )-3ay(2, ; )*y(3. : ) ;y ,3«y (nrooe, : >-3ey!nrooa-l . : )ty!nroea-2 . : )] ; 

69  nrooa  »  nrooa  *  2; 

70 

71  1  loo  interpolate  uaing  computationally  efficient  algorithm. 

72  e2  rn  a.ea;  a3  ■  *.a«2; 

•2  ndx  »  u*onea(m,l)etO:n-l]enrooa; 

74  F  »  y(ndx)  v(-a3*2ea2-a)  ♦  y(ndx»l)  e(3ea3-Sea2*2)  ♦  ... 

•5  y(ndx»2>  •!-3ea3'*4ee2*a)  ♦  y!ndx*3)  e(a3-a2)  . 

76  F  *  F/2. 

78  X  loo  aet  out  of  range  valuea  to  lal 

79  if  nuoutiO,  F(uout)  »  I ale onee (nuout , 1 ) ;  end 

80 

B.2.14  dcubicx.m 


J 

2 

3 

4 

6 

8 

9 
1C 
1 1 
12 
13 
M 

15 

16 
1- 
18 

19 

20 
21 
22 

23 

24 


function  GMcubicxU  , y  ,u) 

XDCUBICX  Derivative  of  Cubic  Interpolation  of  n  1-D  function  e.r.t.  x 

X 

X  G  »  DCUBICX (Y , XI )  roturnn  tha  derivative  of  tha  1-D  function  Y*f(x) 
X  at  the  points  XI  using  cubic  interpolation  length(G)*length(Xl ) 

X  XI  specifies  the  points  originally  used  for  the  interpolation 

X 

X  If  Y  is  of  length  I  then  XI  most  contain  values  between  1  and  1. 

X  The  value  VaV  is  returned  if  this  is  not  the  case 
X 

X  G  *  DCUIICXCX.Y.XI)  uses  tha  vector  X  to  specify  tha  coordinates 
X  for  Y  as  for  ICUBIC 
X 

X  Sea  also  ICUBIC 

X  Clay  B  Thompson  7-18-91 

X  Based  on  "Cubic  Convolution  Interpolation  for  Digital  Image 
X  Processing" ,  Eobert  0  Keys.  IEEE  Trans  on  Acoustics,  Speech,  and 
X  Signal  Processing,  Vol  29,  lo  6,  Dec.  1981,  pp  11 S3- 1160 


if  nargia**2,  X  Vo  X  specified, 
u  ■  y,  y  •  s; 

X  Check  for  vector  problem  If  so,  make  everything  a  column 


4 


4 


« 


a 


e 


a 


e 


e 


•  a  a 


a  a  a 
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24  if  nin(aise(y) )•»! ,  y  ■  y(:).  and 

26  if  min<sise(u))”l  ,  u  •  «<:);  and 

27  [ureas, seals]  •  sixe(y); 

26 

26  alttif  Ut|iaaa3,  S  X  specified. 

40  X  Check  far  vector  problaai.  If  ae,  make  everything  a  colon  vector. 

41  if  min(eisa(y))>al ,  y  »  y(  ),  and 

42  if  min(sixe<x)>»«l,  x  •  x(:>;  and 

44  if  ain(aixa(a)>s<al,  a  »  n(:);  and 

44  [nrous.ncels]  ■  sixe(y); 

46  X  Scala  and  shift  n  to  ba  indicas  into  Y 

46  if  <min(aixe(x)>'al) ,  errorOX  auat  ba  a  factor  '),  and 

47  x  »  x(  : ) ; 

46  [m.n]  *  aixa(x): 

49  if  a  '■  nroas , 

40  error! 'The  length  of  X  auat  aatch  tha  nuabar  of  root  of  Y  ’), 

41  and 

42  u  •  1  ♦  (u-x(l) )a((nroaa-l)/(x(a)-x(l) )) ; 

44 

44  alsa 

46  arror( 'Wrong  nuabar  of  input  arguaants  ’ ) ; 

46  ond 

47 

46  if  nrossO,  arrorCY  auat  have  at  laaat  3  raws.');  ond 
49  [a,n]  «  sixa(u) ; 

60  if  n»l,  u  ■  u*ones(l ,ncols) ;  [a,nj  *  sixain);  and  X  Expand  u 

61  if  n*«ncols,  arrorCTha  nuabar  of  coluaaa  in  XI  and  Y  auat  aatch.’),  and 

64  X  Chack  for  out  of  rango  valuas  of  u  and  not  to  1 
64  uout  ■  f ind( (u<l) I (»>nrows> ) ; 

56  nuout  •  laagth(uout) ; 

56  if  nuoutX),  u(uout)  *  onos( nuout ,1) ;  and 

57 

56  X  Interpolation  paraaatara 
59  a  »  <u  -  floor(u) ) ; 

eo  u  *  floorCu) ;  ,  ...... 

61  d  *  find(uonroua) ;  if  longth(d)X>,  u(d)  *  u(d)-l;  a(d)  ■  i(d)+l;  ond 

62 

64  X  Expand  y  ao  interpolation  ia  valid  at  tho  boundary. 

64  y  •  [3»y(l , :  )-3»y<2, :  >*y(3, : )  ;y  ;3ay(nroaa, :  )-3*y(nrona-l , :  >*y(nross-2 ,  : >]  ; 

65  nrons  •  nrova  *2, 

66 

67  X  loo  interpolate  using  coaputationally  efficient  algoritba. 

66  s2  «  a  as; 

69  ndx  a  u»onee(n,l)a[0:n-l]anroas ; 

ro  a  *  f(ndx)  •(.- 3vs2*d»s-l>  *  j(nix*V  •  (9ea}-10es)  ♦  ... 

71  y(ndx*2)  •(-9«s2**as+l)  ♦  y(ndx*3)  a(3aa2-Jaa); 

72  0  *  C/2. 

74 

74  X  Ion  sat  out  of  range  values  to  lal 

75  if  nuout>0,  G(uout)  *  lafaonaa(nuout , 1 ) ,  and 

76 

B.2.15  dcubicz.m 

1  function  gadcubicy(x ,y ,u ,dJdf ) 

2  XDCUBIC  Darivativo  of  1-D  cubic  interpolation  art  Y 
4  X 

4  X  C  •  DCUBICY1Y ,X1 ,dJdF>  confutes  tho  darivativo  of  tha  cost 

5  X  function  J(F)  uith  respect  to  tho  underlying  variablas  Y  Tho 

6  X  uatrix  dldF  is  the  darivativo  of  tha  cost  function  vith  raspect  to 

7  X  the  interpolated  value  F  •  ICUBIC(Y,I1)  Xl  specifies  tha  points 
6  X  originally  usod  for  tho  interpolation 

9  X 

10  X  G  ■  DCUilCYCX ,Y,X1 ,dJdF)  uses  tha  vector  X  to  specify  tho 

11  X  coordinates  for  Y  as  for  ICUBIC 

12  X 

14  X  Sea  also:  ICOBIC 

14 

15  X  Clay  H.  Thoapson  7*12-91 

16 

|7  if  nargin*a3,  X  to  X  specif iad. 

16  dJdf  «u;  u«y;  y*x, 

19  X  Chock  for  voctor  problem.  If  ao,  make  everything  a  coluan  vector 

20  if  nin(aixa(y))>al ,  y  •  y(:>;  and 

21  if  min(aixe(u))nl ,  u  •  u(:>,  and 

22  Cnroas.ncola]  ■  sisa(y); 

24 

24  alsaif  narginaM,  X  X  specif  iad. 

26  X  Chack  for  vector  problem.  If  so,  oak  a  everything  a  column  vector. 

26  if  min(sixe(y>  )"l ,  y  •  y(  >.  and 


•  • 


•  •  •  • 
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27  if  aia(siza(z) >“1 ,  x  ■  *(:);  ni 
it  it  aia(size(a)>aal,  a  •  u().  and 

29  Carons, acols]  *  sizo(y); 

30  X  Seal*  aad  shift  a  to  ba  indicaa  into  Y 

31  if  (min(size(x))"»l) ,  error (’X  anat  ba  a  tractor.’),  aad 

32  *•»(:); 

33  [a, a]  ■  sized); 

3«  if  a  *•  areas , 

35  error ( 'The  length  of  X  Bust  Batch  the  number  of  roes  of  Y.  ’); 

36  aad 

37  u  »  1  ♦  (u-x(l))e((nross-l)/(x(m)-xd))> ; 

36 

39  alsa 

to  error ( ’Wrong  noaber  of  input  arguaents . ’ ) ; 

4 1  aad 

42 

43  if  nroaaO.  errorCY  aust  have  at  least  3  rose.’);  and 

44  [a. a]  •  siza(a) ; 

43  if  n““l ,  u  •  u*onas(l  .ncols) ;  ta.n]  *  siza(n) ;  aad  X  Expand  a 

46  if  aajr(size(u)~*size(dJdf ) ) ,  error ( 'dJdF  aad  XI  aust  ba  the  saaa  size  ’);  and 

47  if  n’encols,  error (’The  nuaber  of  coloaaz  in  XI  aad  Y  Bast  Batch. ’);  aad 

48  if  a<2,  X  Expand  a  (so  suae  vork)  with  aa  oat  of  range  value. 

49  u  •  [u;zeros(l ,n)] ;  [a, a]  •  siza(n) . 
so  dJdf  “  [dJdf , zeros (1 ,  n)] ; 

51  and 

52 

53  X  Check  for  oat  of  range  values  of  a  end  set  to  1 

54  aout  •  f ind( (u<l ) I (u>nrovs) ) ; 

55  nuout  B  length(uout) ; 

56  if  nuoat>0,  u(uoat)  •  ones (nuout , 1 ) ;  and 

57 

58  X  Interpolation  paraaatars 

59  a  •  (u  -  floor(u)); 

60  a  •  floor(u); 

61  d  ■  f ind(u«*nroos) ;  if  langth(d)X),  u(d)  *  u(d)-i ;  s(d)  •  e(d)+l,  and 

62 

63  X  Compute  teras  for  gradient 

64  s2  ■  s . as ;  s3  »  t  es2; 

65  tO  •  dJdf . a(-n3a2as2-s) ; 

66  tl  »  dJdf . a(3as3-5as2+2) ; 

67  t2  «  dJdf .e(-3es3+«ea2+s) . 

68  t3  »  dJdf  «(s3-s2) ; 

89  clear  a  s2  s3 

70 

71  X  Sat  oat  of  range  terms  to  zero. 

72  if  nuout>0, 

73  tO(uout)  •  zeros (nuout ,1) ; 

74  tl(uout)  •  zeros (nuout , 1 ) . 

75  t2(uout)  ■  zaros (nuout , 1 ) . 

76  t3(uout>  a  zeros (nuout , 1 ) . 

77  dJdf(uout)  a  zaros (nuout , 1 ) , 

78  and 

79 

ao  X  Fora  bubs  for  aach  z  value. 

81  tOsua  ■  zerosfnroea .ncols) , 

82  tlsua  >  zeros (arose , acols ) , 

83  t2saa  a  zeros (areas, ncols) , 

84  t3sua  ■  zaroa(aross, ncols) ; 

85  for  kal:nross, 

86  alaa  •  find(u"«k); 

87  te4sp  a  zsros(a.n). 

88  teop(elea)  ■  tO(alaa) ;  tOsua(k,  )  a  sua(taap) , 

89  taap(alaa)  •  tl(elem);  tleua<k,  )  •  sua(taap); 

90  taap(alaa)  •  t2(alaa);  t2sua(k,:>  a  soa(taap)  . 

91  teap(alaa)  ■  t3(elea)  ;  t3aua(k,.)  •  soa(taap)  , 

92  and 

93  clear  tO  ti  t2  t3 

94 

95  X  Add  terms  froa  boundary  conditions 

96  tlsuad,:)  a  tlsuad,:)  *  3et0sua(l ,  :  )  ; 

97  t2sua(l , : )  •  t2sua(l,  )  -  3et0sua(l ,  ) . 

98  t3sua(l,.>  ■  t3sua(l ,  : )  ♦  tOeuad,.): 

99 

100  t2sua(a-l,:)  *  t2sua(a-l,:)  ♦  3et3som(a-l , : > ; 

101  tlsua(a-l,)  ■  tlsua(a-l,)  -  3et3s«m(m-l , : > ; 

102  tOsaa(a-l , : )  •  t0eua(a-l,  )  *  t3sua(a-i , : ) , 

103 

104  X  Ion  coabine  to  fora  gradient. 

105  g  •  zerostnroes .ncols) . 

106  g(l,  )  a  t0sua(2,;>  *  tlsuad,:); 

107  g(2 , : )  ■  t0sua(3,:)  *  tlaua(2,:)  *  t2oua(l ,) . 

108  for  ka3:arous-l, 

109  g(k,  )  •  tOeaa(k*l ,  )  *  tlsua(k,.)  ♦  t2eua(k-l.)  *  t3sua(k-2 , : ) , 
no  end 


•  ••••• 


Appendix  C 

Test  Surface  Descriptions 


This  appendix  contains  information  on  how  to  create  the  crater  and  hill  test  surfaces 
and  associated  test  images. 


C.l  Crater  Surface 

The  crater  on  the  flat  plane  is  formed  by  intersecting  two  spheres,  one  with  radius 
10  and  the  other  with  radius  9.48,  with  a  flat  plane.  The  routines  makepair4  and 
crater_depth  were  used  to  create  the  crater  images. 


C.1.1  makepair4.m 

1  X  Dak*  atarto  pair  (this  i(  tha  bait  routina) 

2  claar 

3  DEPTH  a  'cratar.dapth ' 

a  (DEPTH  *  ’ aphara.dapth 1 

5  XGHAD  a  'cratar_grnd ’ 

6  n  •  AS:  X  Tba  ilia  of  iaaga. 

•  au  «  1,  X  Optiaixation  paraaatar 
8 

9  b  a  SOO; 

10  f  ■  -1000 ;  X  Ha  ara  larking  in  tha  aaaa  coerdiaataa  aa  tha  itarf aca  far  i  1  y 

11 

12  X  Caaara  coerdinata  calibration 

13  *0  a  aaal ( [DEPTH,  •  (0.0)  ’]) 

14  vO  ■  0; 

15  »1  •  *0  ♦  baf/xO/2 

16  *2  «  »0  -  baf/nO/2 

17 

16  gam  •  fab/2, 

19  dalta  »  nO/f: 

20  if  1  X  Cana  a  {Hard  caaa> 

21  pal  ■  1;  qal  a  1. 

22  pa2  •  1;  qa2  •  .  1 ; 

23  prafix  ■  t’cr’ ,int2atr(n) , ’a’] ; 

24  alaa  X  Caaa  b  {Sony  caaa> 

25  pal  «  .2;  qal  ■  -.5; 

26  pa2  •  -  3;  qa2  »  1 , 

27  prafix  a  [>cr’,lnt2atr(a)  ,’b’]  : 

28  and 

2»  Xpal  *  S:  qal  »  -.6: 

30  Xpa2  ■  5.  qa2  •  -.6; 

31  Xpal  ■  0;  qal  ■  0; 

32 

33  [x.y]  •  da«ain2d(-12:24/(n+l>:12.-12:24/<o*l>12  ); 

34 

35  X  Inaga  paraaatora 
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36 

37 
36 

39 

40 

41 

42 

43 

44 

45 

46 
4  7 
46 

49 

50 

51 

52 

53 

54 

55 

56 
5? 
56 

59 

60 
61 
62 

63 

64 

65 

66 
67 
66 

69 

70 

71 

72 

73 

74 

75 

76 

78 

r9 

80 

81 

62 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 

95 

96 

97 
96 
99 

100 

101 

102 

103 

104 

105 

106 

107 

108 

109 

110 
111 
112 

113 

114 

115 


doltax  *  x(l,2)-x<l.l); 

puiM  *  [f /doltax  X  Focal  length  for  unity  spacing  grid 

b  X  basolino  distance 

sO  X  Calibration  dapth 

pal ;qal ;pa2 ;qs2  X  Light  positions 

(x(2,2)+vl) /doltax- 1  X  x-calibration  as  offsot  froai  E(l,l). 
y(2,2)/doltax-n  X  y-calibrat ion  as  offsot  fro*  E(l,l). 
(x(2,2)*v2)/doltax-i  X  x-calibration  as  offsot  fro*  E(l,l) 
y(2,2)/doltax-n  X  y-calibration  as  offsot  froai  8(1,1). 

1  X  Grid  spacing 

dol  ■  zO*(max(x(: >)-*in(x< : )>)/f/(n+l) ; 


z  »  oval( [DEPTH, * (x,y) '] ) ; 

•cp*sh(z  ,dol) 
titloPTruo  Surfaco’) 

X  Dotormino  dopth  function  for  loft  isago  (1) 


orr  •  1;  ■  *  0; 
x*  *  zO*n onos(x); 
gMM/l0 

vhilo  (orr>  1  o-S)i(m<2S) , 

stomp  •  oval ([DEPTH, ’  (x+vl -gamma . /zl-vO.y) 
orr  *  sum(  (stomp ( : )-zl< : ) ) . *2  ) 
zl  »  Zl  ♦  ■u*(ztomp-zl > ; 
m  *  m  ♦  1; 
ond 

o^osh(zl  .dol) 
titloC’Loft  Surfaco’) 


’]  > ; 


X  dotormino  dopth  function  for  right  imago  (2) 
orr  *  1 ;  m  »  0 ; 
z2  *  zOomonos (x ) ; 
shilo  (orr>l  o-5)i(m<25) , 

ztomp  »  oval ( [DEPTH, »<x*v2+gamma./x 2 -vO.y) ’]) ; 
orr  »  sum(  ( ztomp ( : )-z2( : ) )  “2  ) 
z2  *  *2  ♦  nno(xtomp-*2) ; 
m  »  ■  ♦  1; 
ond 

o<piosh(x2  .dol) 
t it lo(’ Right  Surfaco’) 


hx  -  l~l  O  1 1/2; 
hy  ■  Ll;0;-l]/2; 


root 

cols 


2 : n**l ; 
2  n*l ; 


X  Eight  Imago 

zx  *  filtor2d(zl.hx. ’rosizo ' )/doltax, 
zy  ■  filtor2d(xl.hy, ’rosixo*) /dolt ax; 

pi  ■  f*xx(rovs, : ) . /((x(roos ,cols>*vl>  •z*(rowa ,  )*zl (ross ,cols>) ; 
ql  -  f*sy( cols) • /(y(rovs , cols)  *xy( : ,cola)+zl(roos , cols)) ; 

El  *  map  (pi  ,ql  .psl.qsl) ; 

X  Loft  Imago 

zx  *  f iltor2d(x2,hx, 'rosizo ’) /doltax; 
zy  *  filtor2d(x2.hy, ’rosizo ’) /doltax 

p2  *  f •zx(rovs, : ) . /( (x(roos .cols )+v2)  *xx(roos . ; )oz2(roos .cols) ) ; 
q2  *  f*xy( : .cols) ./(y(roos , cols)  •xy( : ,cols)*x2(roos .cols) ) ; 

B2  »  rmap(p2,q2,ps2,qs2) ; 

X  Contor  Global  coordinato  systom 
zx  •  filtor2d(x,hx, ’rosixo’ ) /dol tax, 
zy  ■  f iltor2d(x,hy . ’rosixo • ) /dol tax; 

p  ■  f*xx(rovo, :) ./(x(rovs, cols) - *xx(roos )+z(rovs, cols) > ; 
q  *  f»xy( : .cols) ./(y(ro»s,cols)  szy(: ,cola)+z(ro»s,cols>) . 

E  ■  map(p  ,q,ps2  ,qs2)  ; 

clg. 

subplot (121) ,  o^osh(zl  ,dol) , 
subplot  (122) ,  o^osh(z2  ,dol) 
subplot (111)  ,  tltls(’lasgo  contours') 

ais«(o([Il  ,zoros(a,l)  ,12]  ,  [0  l]  ) 

clour  pi  ql  p2  q2  zx  17  p  q  E  tuui  orr  ■  n  ztomp 

clour  psl  ps2  qsl  qs2  zO  bO  b  f  doltu  hz  bp  ross  cols  *1  *2 

clour  shi  rO  z  jr  dsltuz 


o 


o 


o 


o 


o 


o 


o 


o 


•  •  o  0  0 
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C.  1.2  crater-depth,  m 

1  function  [x,rho]»dopth(x,y> 

2  XCkATEl.DIPTX  Dopth  function  for  crntor  on  x  flat  piano. 

3  X 

4  X  Z  •  CtATKl.  DEPTH  (X,Y) .  X  and  Y  nuxt  bo  of  tha  tana  aixa. 

5  X 

6  X  Cratar  is  dafinad  batsaan  -10  <»  x,y  <*  10. 

a  xO  «  -1000; 

9  rl  ■  10;  r2  «  9.46; 

10  dl  •  find(x.*x  ♦  y.sy  <»  0.75«rl»rl>; 

11  d2  «  find(x.*x  ♦  y.ay  <*  0.7S»r2*r2); 

12 

13  [«,n]  *  xixa(x) ; 

14  z  •  xO*onax(ai,n) ; 

is  z(dl)  «  x(dl)  ♦  xqrt(rl»rl  -  x(dl)  »x(dl)  -  y(dl>  ay(dl))-5, 

16  x(d2)  »  nin(x(d2) ,  x0+12-xqrt(r2»r2  -  x(d2).*x(d2>  -  y(d2)  «y(d2) ) ) ; 

17 

18  Xx  »  xO  ♦  (x-x0)/2;  X  fcaduca  haight  by  too 

19 

20  if  nargoutil , 

21  X  Xlbado  (rho) 

22  rho  •  aonas(x); 

23  d  *  find(abx<x-y><3> ; 

24  rho(d)  «  0.7aaono*(d) ;  X  Dark  xtrip  aeroax  diagonal 

25  and 

C.2  Hill  Surface 


A  fractal  based  method  was  used  to  create  the  underlying  data  matrix  for  the  hill 
surface.  Given  this  underlying  data  matrix  the  hill  surface  is  formed  using  bicubic 
interpolation  to  define  the  surface  points.  The  data  for  the  hill  is  shown  in  the  matrix 
(DATA_MATRIX)  below. 


-  1000 


0.7210 

1.0934 

1.1456 

0.7672 

0.5582 

0.6868 

1.0628 

0.9350 

0.4301 

1.2280 

0.4993 

0.5555 

0.8574 

0.3552 

0.8864 

1.0791 

0.2720 

0.5742 

0.6650 

0.3307 

0.2967 

0.7894 

0.2520 

0.5136 

0.2594 

1.1026 

1.0038 

0.4796 

0.8690 

0.4103 

0.3271 

0.5276 

1.0706 

1.1861 

0.4380 

1.1673 

1.1548 

0.3879 

0.2889 

0.3674 

0.8726 

0.7950 

1.3493 

0.4233 

0.3691 

0.2568 

0.4342 

0.6595 

0.5902 

(C.l) 


The  routines  makepair_data  and  data_depth  were  used  to  create  the  hill  images. 
The  mountain  images  were  formed  using  a  similar  method  based  on  a  33-by-33  data 
matrix  that  is  too  large  to  be  presented  here. 


C.2.1  makepair-data.m 

1  X  Haka  ataroo  pair  baxod  on  data 

2  cloar  X  claar  all  data 

3  global  DXTA.HXTtlX 

4  DEPTH  *  ’data.dopth’ 

5  n  -  65  X  Tha  sirs  of  iaags . 

6  nn  *  n-4-2,  X  Tha  xixo  of  x. 

7  no  »  1 ;  X  Optiaixation  paraaator 
a 

9  if  1 .  X  Hrinklod  turfaco  (hill) 

10  load  data_aatrix 
n  b  *  500, 

12  pal  »  -1 ;  qxl  ■  1 ; 

13  px2  ■  3;  qx2  •  .1 , 

14  prafix  •  [  ,int2«tr(n) ,  'c ’] ; 
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15 

16 
17 
38 

19 

20 
21 
22 

23 

24 

25 

26 
27 
26 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 
46 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 


72 

73 

74 

75 

76 

78 

79 

80 
81 
82 

83 

84 

85 

86 
87 
86 

89 

90 

91 

92 

93 

94 

95 
98 


alsa  1  Lafta  (aouatain) 
load  lafta.data 

*0  •  -1000; 

DATA. MTU X  »  DATA. RATII I  ♦  xO; 
b  •  100; 

psl  ■  5;  qxl  *  5; 

ps2  *  -  S;  qs2  »  0; 
prafix  ■  ['I** ,int2str(n) , *b»] ; 

•ad 

[ad.nd]  ■  siza( DATA. MATRIX)  ; 

f  *  -1000 ;  X  Va  art  working  in  tha  >aaa  coordiaatas  ax  tha  surfaca  for  x  ft  y 

X  Camara  coordinata  calibration 
xO  -  aval ( [DEPTH ,’(0,0)']) 
vO  *  -baf/xO/2; 
vl  »  vO  ♦  baf/xO/2 
v2  »  vO  -  b*f/z0/2 


gaaaa  *  f«b/2; 
dalta  *  xO/f ; 

[x,y]  -  doaain2d<[l:(ad-l)/(nn-l)ad3-(ad*l)/2,[l:(nd-l)/<nn-l):nd]-(nd*l>/2) ; 
x  *  x*  9;  y  *  y*.9;  X  Caatar  90X  of  matrix 


X  Iaaga  paramatars 

dal tax  -  x(l,2)“x(l,l); 

para ms  *  [f/daltax  X  Focal  langth  for  unity  spacing  grid 

b  X  basaliaa  diftanca 

xO  X  Calibration  dapth 

pal ;qsl ;ps2 ;qs2  X  Light  positions 

(x(2,2)+vl)/daltax-l  X  x-calibration  as  offsat  from  (1,1). 
y(2,2)/daltax-n  X  y-calibrat ion  as  offsat  from  (1,1). 
(x(2,2)+v2)/doltax-l  X  x-calibration  as  offsot  from  (1,1). 
y(2,2)/daltax-n  X  y-calibration  as  offsat  from  (1,1). 

1  X  Or id  spacing 

dal  *  zO*(aax(x< : ))-ain(x( : )))/f/(n+l) ; 

z  -  aval ( [DEPTH , ’ (x , y )  ’  ]  ) ; 
aqaash(z,dal) 

XaMsh(x) 

titlaCTrua  Surfaca') 

X  Dataraina  dapth  function  for  laft  iaaga  (t) 


arr  *  1 ;  a  ■  0; 
zl  *  zO*aw»nas(x)  ; 

gamma/ zO 

vhila  (arr>  1  a-5)ft(a<25> , 

xtaaip  *  aval ( [DEPTH ,  *  (x+vl-gamma 
•rr  *  aua(  (ztaaip(  :  )-zl( : ) ) .  “2  ) 
zl  ■  zl  4  aua(ztamp-zl) ; 
a  •  m  ♦  1; 
and 

aqaash(zl , dal ) 
titla(’Laft  Surfaca') 


/zl-v0,y) 


’3); 


X  dataraina  dapth  function  for  right  iaaga  (2) 
•rr  »  1 ;  a  ■  0; 
z2  *  zOaaonas(z); 
vhila  (arr>l .a-5)ft(a<25) , 

ztaap  •  aval ( [DEPT! , * (r+v2+gamma. /z2-v0,y) ’] ) ; 
•rr  •  aua(  (ztaap( : )-z2( : ))  “2  ) 
z2  ■  z2  ♦  au*(xtaap-z2) ; 
a  ■  a  ♦  1; 
and 

a^ash(x2  ,dal  ? 

titla( 'Right  Surfaca') 


hz  ■  [-1  0  ll 
hy  ■  [1 ;0; -1J 


rovs 

cols  ■ 


2n*l; 
2 : n*t ; 


n, 

12. 


X  Right  Iaaga 

xx  •  f iltar2d(xl ,hx , 1 rasixa ’ ) /daltax ; 
zy  ■  f iltar2d(zl ,hy, ’rasixa ») /daltax; 

pi  •  f  «xx(rovs , : )  /( (x(rova  .cols^vl)  azx(rovs  , :  )+zl(rovs  .cols) ) ; 
ql  »  f*zy( : ,cols)  /(y(rovs, cols) .  ozy( . ,cols)+zl (rovs, cols)) . 

El  a  raap(pl ,ql ,psl ,qsl) ; 


a 


a 
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9-  X  Left  Image 

98  zx  ■  filter2d(x2,hx, ’retixe * )/deltex , 

99  xy  *  tilter2d(z2,hy , 'retixe *)/deltex; 

100  p2  *  f*xx<ro»«, . ) . /((xCroo* .col*)+v2)  *xx(roat ,  >+z2<rov* ,col*> ) ; 

101  q2  *  fexy(- .cola) ./<y(ro*a, colt) .*zy( : ,colt>*z2(rowt, colt)) ; 

102  E2  *  naap(p2,q2 ,p*2,q*2) ; 

103 

104  X  Center  Global  coordinate  ayatem 

105  zx  *  f ilter2d(z.hx, ’raxixa * )/daltax ; 
l Ob  zy  ■  1 iltor2d(x ,hy  ,  ’retix* ' )/d«ltax ; 

io'  p  *  f*zx(rova /(x(rovs, cola) . oxxCroxa )+z(rova , colt) ) , 

10*  q  *  f»zy< colt) ./(yCrota.cola) . *zy( : , colt) ♦*( rot* , colt) ) ; 

109  E  ■  map(p,q,pt2,qt2) ; 

no 

ill  clg.*ubplot(221) 

U2  tubplot (121) ,  eqmeah(zl .del) 

113  tubplot  (122) ,  e<|Beah(z2 .del) 

114  aubplot(lll) ,  title (’Image  contour*') 

us 

116  aiaage([El ,xerot(n,l) ,E2] , [0  1]) 

li? 

li*  clear  pi  ql  p2  q2  zx  xy  p  a  E  xataut  err  a  n  zteap 

119  clear  ptl  p*2  qtl  q*2  xO  bO  b  I  delta  hx  hy  rota  colt  vl  v2 

120  clear  mu  vg  x  y  ad  nd  nn  deltax 

121 

12?  ztrue  *  z; 

123 

C.2.2  data.depth.m 

l  function  [z ,rho]*dapth(x  ,y ) 

1  XDATA  DEPTH  Dopth  function  bnnod  on  data  matrix. 

3  X 

.1  Z  •  DATA_DEPTH(I ,Y>  X  and  T  nuat  ba  of  tho  a  ana  aizo. 

b  X 

6  X  [Z.AHO]  »  DATA_DEPTH(X ,Y)  roturna  albado  also. 

7  X 

8  X  Kalian  on  tho  global  DATA.PATAIX  to  dafino  aurfaca 

9  X 

10  X  For  DATA. Unix  n-by-n ,  valuaa  in  X  nuat  ba  bataaan  -(n-l)/2  and  (n-l>/2, 
n  X  valuaa  in  Y  must  ba  bataaan  -(»-l)/2  and  (»-l)/2. 

13  Xglobal  DATA. Unix 
M  [n.n]  «  aiza(DATA.uniX) . 
lb  [xx.yy]  •  donain2d(l : n , 1 :■) ; 

16 

1-  a  *  I  ♦  (ar»l)/2;  y  ■  y  *  <n*t>/2; 

18 

19  d  >  find(floor(xXl) ;  x(d)  »  onaadangth(d)  ,1)  ; 

20  d  »  find(x>n),  x(d)  »  naonandangth(d) , 1 ) . 

11  d  *  find(floor(y)<l) ;  y(d)  »  onaa(langth(d) , 1 ) ; 

22  d  *  find(y>aO  ;  y(d)  *  naonaadangthCd)  ,  l ) ; 

23 

2.  z  «  bicubic(xx,yy  ,DATA.uniX,x,y) . 

2b  Xz  >  blinaar(xx,yy,DATA_UniX,z,y>; 

26 

27  d  *  finddanan(z) )  , 

28  if  langth(d)>0,  kayboard,  and 

29  if  langth(d)>0,  z ( d)  •  onaadangth(d)  ,  1 )  •oun(DATA_UniX(  : ) )  ;  and 

30 

li  if  nargoutil, 

32  Xd  •  f ind(aba(x-yXl) ; 

33  zO  ■  nin(x( :  > )  ; 

34  d  *  find((z>.6*zO)  A  (z<  T*z0)). 

3b  rho  ■  aonaa(x) ; 

36  rho(d)  ■  0  7*aonaa(d). 

3*  and 
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